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1. Introduction 

1 . 1 . 

Let F be a non-Archimedean locally compact field of residual characteristic p, let G be the 
F-points of a connected reductive group over F together with a closed subgroup H of G, and let 
R be an algebraically closed field of characteristic different from p. Given irreducible smooth 
representations n of G and a of H with coefficients in R, it is a question of general interest in 
representation theory, known as the branching problem, to understand whether n restricted to 
H has cr as a quotient. If R is the field of complex numbers, this question is classical and well 
understood is many situations (see for instance [6, 7]). A case of particular interest is when a 
is the trivial representation. In this situation, the representation ir is said to be H-distinguished 
if its restriction to H has the trivial representation as a quotient, that is, if n carries a nonzero 
H-invariant linear form. 
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1 . 2 . 


In this article, we are interested in the case where G is the general linear group GL n (F), with 
n ^ 2, and H is the group GL n _i(F) embedded in G via: 


x 0 
0 1 


When R is the field of complex numbers, it is a consequence of a result of Waldspurger [ 25 ] 
that, for n = 2, any infinite dimensional irreducible representation of G is H-distinguished. The 
classification of all H-distinguished irreducible representations of G for n = 3 has been done by 
D. Prasad [ 15 ]. For any n 5= 2, Prasad [ 15 ] has also proved that any generic representation of G 
has every generic representation of H as a quotient, and Flicker [ 5 ] classified all H-distinguished 
irreducible unitary representations of G. The classification of all H-distinguished irreducible rep¬ 
resentations of G for any integer n ^ 3 has been obtained by Venketasubramanian [ 19 ], in terms 
of Langlands parameters. Thus, when R is the field of complex numbers, the question is well 
understood. In this paper, we investigate the case where the field R has positive characteristic 
t different from p. 


1 . 3 . 

The representation theory of smooth representations of GL n (F) with coefficients in any alge¬ 
braically closed field R of characteristic £ ^ 0,p has been initiated by Vigneras [ 21 , 22 ] in view 
to extend the local Langlands program to representations with coefficients in a field (or ring) as 
general as possible (see for instance [ 23 ]). It has then been pursued by Dat, Mfnguez, Stevens 
and the first author [ 4 , 11 , 12 , 13 , 14 , 16 ], In many aspects, it is similar to the theory of 
complex representations of this group: the fact that l is different from p ensures that there is 
an R-valued Haar measure on G, the functors of parabolic induction and restriction are exact 
and preserve finite length, there is a theory of derivatives, there is a notion of cuspidal support 
for irreducible representations and a classification of these representations by mutisegments. 
However, there are also important differences: the measure of a compact open subgroup may 
be 0, and the notions of cuspidal and supercuspidal representations do not coincide, that is, 
a representation whose all proper Jacquet modules are zero may occur as a subquotient of a 
proper parabolically induced representation. The combinatorics of multisegments is also much 
more involved, since the cardinality q of the residue field of F has finite order in R x . 


1 . 4 . 

We now come to the main theorem of this article. Let R denote an algebraically closed field 
of characteristic different from p (possibly 0) and write e for the order (possibly infinite) of q in 
R x . Write v for the normalized absolute value of F giving value q -1 to any uniformizer. Let us 
fix a square root of q in R, denoted ^fq. Given integers k e Z and n ^ 1, we write: 

I# 2 : g ~ ( v ^)-fc-vai(det( 9 )) 

where val is the normalized valuation on F and det is the determinant from G to F x . If n, a are 
smooth representations of GL U (F), GL„(F) respectively, with u + v = n, we denote by 7r x u the 
normalized parabolic induction of 7r ® a to G along the standard (upper triangular) parabolic 
subgroup. When e > 1, the induced representation: 

(1-1) V n = x „ 


(n+l)/2 
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has a unique irreducible quotient, denoted A n (see Example 4.3). Note that, when e divides n, 
this representation is the trivial character. Let us write l n for the trivial representation of G. 

Theorem 1.1. - Suppose that n ^ 2 and e > 1. An irreducible representation of GL n (F) is 

GL n _i(F )-distinguished if and only if it belongs to the following list: 

(1) the trivial representation l n ; 

(2) an irreducible representation of the form n~_\ x x with x « character o/GLi(F); 

(3) an irreducible representation of the form l n -2 x t with r an infinite dimensional irreducible 
representation o/GL 2 (F); 

(4) the representation A n and its contragredient. 

As in the complex case, the proof of Theorem 1.1 is by induction on n. There are two parts 
to the proof of Theorem 1.1: proving that the representations in the list offered by the theorem 
are H-distinguished is the easier part. The more difficult part is to show the converse, namely 
that all irreducible representations which are H-distinguished are in the list. 

1 . 5 . 

Since our proof is by induction, we first treat the case when n = 2 and obtain a classification 
(see Theorem 3.8) of all the GLi(F)-distinguished irreducible representations of GL 2 (F). When 
e is not 1, the result turns out to be the same as in the complex case: all infinite-dimensional 
irreducible representation of GL 2 (F) are distinguished and their space of GLi(F)-invariant linear 
forms has dimension 1. When the characteristic of R divides q — 1 however, this dimension is 2 
for certain representations. 

1 . 6 . 

Assume now that n ^ 3. As in the complex case, one can show by restricting to the mirabolic 
subgroup that none of the cuspidal representations of G are distinguished (Theorem 8.2). Since 
any non-cuspidal irreducible smooth representation of G is a quotient of a parabolically induced 
representation of the form o x t with c, r smooth irreducible representations of GL U (F), GL„(F) 
for some integers u, v ^ 1 such that u + v = n, it is natural to study the distinction of o x r. 
This was carried out in [ 19 ] in the complex case. In the modular case, it works as in the 
complex case: one gets a set of three necessary conditions for this induced representation to 
be H-distinguished, of which two are sufficient (see Lemma 8.8). This is attributed to the exis¬ 
tence of three orbits for the action of H on the homogeneous space made of all subspaces of 
dimension u in F n , out of which two are closed. The induced representations in (2) and (3) of 
Theorem 1.1 are shown to satisfy one of the sufficiency conditions coming from Lemma 8.8 (see 
Corollary 8.12). The contragredient of A n , when non-trivial, is realized as a subrepresentation 
of a distinguished principal series of length 2, the quotient of which is a nontrivial character and 
is non-distinguished (see proof of Lemma 8.11). 

1 . 7 . 

To prove the converse of Theorem 1.1, we first prove that any H-distinguished representation 
of G is a quotient of a representation of the form p x x where p is an irreducible representation of 
GL n _i(F) and y a character of F x . In particular, when e > 1, such a quotient is unique. Using 
the conditions of Lemma 8.8 mentioned above and the induction hypothesis, we can specify p 
and x f° be in a list (see Proposition 8.17). Then, when e > 1, we analyze the unique irreducible 
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quotient of all these p x x- We show that if the quotient is distinguished, then it must be in our 
list. The case when e = 1 presents additional difficulties which we shall touch open below. 

1 . 8 . 

We now describe the contents of the article. In Section 2 we set some basic notation, and deal 
with the case n = 2 in Section 3. The complete classification for n = 2 is obtained in Theorem 

3.8. We begin Section 4 by recalling some general results on t'-modular representations of GL n (F) 
from [ 21 , 13 ], We get a complete description of the subquotients of representations of the form 
Z(A) x Z(A') where A, A' are segments and A' is of length at most 2 (see Propositions 4.10 and 
4.13). In particular, Proposition 4.10 may be deemed to be a generalization of [ 20 , Theoreme 3]. 
Moreover, comparing with [ 26 , Proposition 4.6], Propositions 4.10 and 4.13 highlight one of the 
essential differences between principal series representations in the complex and modular cases: 
a product of two characters has length at most 2 in the complex case, a fact which does not hold 
as such in the modular case. The representation A n , which plays a essential role in the article, 
is defined in Example 4.3 for e > 1, and in Definition 5.4 in general. More generally, Section 5 
is devoted to the case where e = 1. The avatar U n of A n is defined in Example 4.11. In Section 
6, we compute the derivatives of A n and II n . 

In Section 7, we prove a criterion for irreducibility of a product of the form Z(A) x L(A') where 
A' has length 2. This is a modular version of a result known in the complex case (Theorem 3.1 
in [3]). We begin Section 8 with some basic results on H-distinguished representations of G. The 
first tool is to use Lemma 8.8, the conditions that we get from the three orbits that we mentioned 
above. This, along with some of its consequences, yields us Proposition 8.17 and we get a list 
of representations of the form p x X ( see the list following Proposition 8.17): understanding the 
distinction of the quotients of representations in this list proves the difficult part of Theorem 1.1. 
The second tool in our proof is Proposition 8.7 using the Bernstein-Zelevinski filtration, which 
was available for the complex case [ 5 , 15 ] and holds for R. The computation of the quotients 
of p x x in the list obtained from Proposition 8.17 is the content of Sections 9-12. 

1 . 9 . 

We now explain some of the subtler ideas behind our proof which is different from the one in 
[ 19 ] proved for complex representations. In [ 19 ], the main tool in analyzing the existence of a 
unique irreducible quotient is the Langlands Quotient Theorem and certain results of Zelevinski 
[ 26 ], When these theorems fail to apply, [ 19 ] uses Theorem 7.1 of [ 26 ]. In fact, we use Propo¬ 
sition 4.2 which is sufficient for us to analyze the representations coming from the Lemma 8.8 
when e > 1. Indeed, if one were to use Proposition 4.2 in the complex case, then the proof of 
Theorem 1.1 in [ 19 ] simplifies to some extent without having to resort to Theorem 7.1 of [ 26 ], 
because there we have to analyze all subquotients of a certain induced representation. 

1 . 10 . 

However, in the modular case, even if Proposition 4.2 guarantees the existence of a unique 
irreducible quotient for the representations p x y arising from Proposition 8.17, to explicitly find 
this quotient is more difficult. This is due to the fact that, in order to determine whether the 
unique irreducible quotient of p x x is in the list offered by Theorem 1.1, we have to realize it 
as a quotient of a larger principal series and this larger principal series may not have a unique 
irreducible quotient. In such a situation, we had but no choice to use the analogue of Theorem 
7.1 of [26] for the larger principal series in hand. For our purposes, we reduce it to understand 
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the subquotients of representations of the form Z(A) x Z(A 7 ) where the segment A 7 has length 
at most 2. These subquotients have certain natural properties (see Section 4, PI to P6) proved 
in [13] which enables us to describe the subquotients. This result is obtained in Propositions 
4.10 and 4.13. We then use Proposition 8.7 to rule out the subquotients in the larger principal 
series which are not in the list of Theorem 1.1. 

1 . 11 . 

Let us mention that, when e = 1, the list in Theorem 1.1 is not exhaustive. The first problem 
is that the representation (1.1) need not have a unique irreducible quotient. In particular, all 
its irreducible subquotients are H-distinguished (see Lemma 8.14), which is different behavior 
when we compare with the case when e > 1. This forces us to consider more representations in 
the list offered by Proposition 8.17 and the tools that we use does not seem to be sufficient to 
understand the distinction of the quotients. 

1 . 12 . 

We now give a few remarks. First, the theory of p-modular representations of p-adic reductive 
groups is very different from the ^-modular theory. This is why we have chosen to focus on the 
case where R has characteristic different from p. 

In the complex case, the pair (G, H) is known to be a Gelfand pair (see [1]). This is no longer 
true in the modular case: when e is equal to 1, some H-distinguished have a 2-dimensional space 
of H-invariant linear forms (see Theorem 3.5 and Remark 12.13). 

When comparing the results in [19] with Minguez [9], the classification of all H-distinguished 
irreducible complex representations of G turns out to be easily expressed in terms of the local 
theta correspondence from GL 2 (F) to GL n (F). It would be interesting to investigate this in the 
modular case, by developing an ^-modular theta correspondence (see [ 10 ]). 
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2. Notation and preliminaries 

In all this article, we fix a locally compact non-Archimedean field F ; we write 0 for its ring 
of integers, p for the maximal ideal of 0 and q for the cardinality of its residue field. We also 
fix an algebraically closed field R of characteristic not dividing q. 

We write e for the order (possibly infinite) of the image of q in R x and define: 

, _ f 0 if R has characteristic 0, 

( the smallest positive integer k ^ 2 such that 1 + q + ■ ■ ■ + q k ~ l = 0 in R otherwise. 

When R has characteristic t > 0, we have / = e if e > 1 and / = t if e = 1. 

Given a topological group G, a smooth R-representation (or representation for short) of G is a 
pair ( 7 r, V) made of an R-vector space V together with a group homomorphism 7r : G —> GL(V) 
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such that, for all v e V, there is an open subgroup of G fixing v. In this article, all representations 
will be supposed to be smooth R-representations. 

A smooth R-character (or character for short) of G is a group homomorphism from G to R x 
with open kernel. 

Given a representation n and a character x °f G, we write nx for the twisted representation 

9 *-> n{g)x{g)- 

For n ^ 1, we write G n = GL n (F), and G n for the set of isomorphism classes of its irreducible 
representations. In particular, Gi will be identified with the group of characters of Gi. 

Given a representation n of G n , n ^ 1 and ^ e Gi, we write it ■ n = 7r(/x o det). If n has finite 
length, we write [ 7 r] for its semi-simplification. 


3. The pair (GL 2 (F), GL^F)) 
Write G = GL 2 (F) and let: 


H = 


x 0 
0 1 


xe F > 


G. 


Let B denote the Borel subgroup of G made of upper triangular matrices, and write: 

o i 6 G‘ 


If X is a locally compact topological space and A is a commutative ring, let C®(X, A) denote 
the space of all locally constant and compactly supported functions from X to A. 

We write dx for the R-valued Haar measure on F x giving measure 1 to the subgroup 1 + p of 
principal units (see [21, 1 . 2 ]). 


3.1. The principal series 

Let «i, a 2 be two smooth R-characters of F x . Let: 

V = V(«i, a 2 ) 

denote the (non-normalized) parabolic R-induction Indj^ai ®cc 2 ), that is the space of all locally 
constant R-valued functions / on G such that f(mng ) = ai(mi)a 2 (m 2 )/(g) for all: 

m = (T m 2 ) G G ’ U G (o l) - 9 e 

which is made into a smooth R-representation of G by making G act by right translations. Write 
W for the subspace of V made of all functions vanishing at 1 and s. The map: 

W^ S“(F X ,R) 

which associates to / e W the function: 

i)) 

is an isomorphism of R-vector spaces, and becomes an isomorphism of representations of H if 
the right hand side is endowed with the action defined by: 

a ■ 4> : x >—> a 2 (a)^)(xa _1 ), i,aeF x . 
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Up to a nonzero scalar, there is on W a unique nonzero H-invariant linear form, given by: 

M = /~J/(.(S a 2 (x)~ 1 dx. 

F x 

Let a denote the character of B extending aq <S) a 2 . Fix an integer i ^ 1 such that 0 : 1,02 
are trivial on 1 + p 4 , and let Kj be the subgroup of GL 2 ( 0 ) made of matrices congruent to the 
identity mod p'. We define two functions /o and on G: 

(1) /o is supported on BsK* and fo(bsx) = a(b) for all b £ B, x £ K*. 

(2) /oo is supported on BK* and fa>(bx) = a{b ) for all b £ B, x £ K*. 

As a is trivial on B n K*, these functions /o, /oo are well defined. They are in V but not in W. 
Lemma 3.1. - Given f e V, there is a unique function w(f ) £ W such that: 

f = f{s)fo + /(l)/oo + w(f). 

This defines a projection w : V —> W with kernel spanned by fo and /qq. 


Proof. — This follows from the fact that s does not belong to BKj. □ 

Let A be an H-invariant linear form on V. It is characterized by A(/o), A(/oo) £ R and its 
restriction to W. As this restriction is H-invariant, it is of the form cpL for a unique scalar c £ R. 

Corollary 3.2. — The space V* H of ^-invariant linear forms on V has dimension ^ 3. 


Now let A be a linear form on V extending y. We search for a necessary and sufficient condi¬ 
tion on A(/o), A(/oo) £ R for A to be H-invariant. By definition, this linear form is H-invariant 
if and only if: 

a ((s l 0' / ) =A(/) 

for all x £ F x and / £ V, and it is enough to check this condition for all x of valuation 1 and 
/ = /o) /oo- Let t £ F x be of valuation 1. We have: 


t 0 
0 1 


fo = u 2 (t)fo + w 


t 0 
0 1 


fo), 


t 0 
0 


^•/oo = arW/oo + w 


Thus the condition writes: 

(1 - a 2 (t))X{fo) = y 0 (t) and (1 - ai(t))\(f x ) = has(t) 
for all t £ F x of valuation 1, where: 


t 0 
0 1 


fo 


and /das (t) = y ( w 


t 0 
0 1 


fa 


ho (t) = hyw 
Lemma 3.3. — We have: 

ho{t) = —ci 2 (t) 1 ~ l J a 2 (x)- 1 dx and has{t) = a\{— l)ai(t) 4 J aq(x) _1 dx. 
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Proof. — Given ieF x , write me Z for the valuation of x (normalized in such a way that any 
uniformizer has valuation 1) and: 

'l x^ 


l(x) = 


0 1 


We have: 

(3.1) 
and: 

(3.2) 

Note that: 


sl(x) = s 


1 x 
0 1 


e BsKi 


m ^ i 


sl(x) = 


—x 


-l 


0 x 7 V x 


1 0 

,-i i 


e BK, m ^ —i. 


si(x) 


t 0 
0 1 


= s 


1 x\ ft 0 
0 1/ VO 1 


We have: 


Mo(t) = 

F x 

Poo (t) = J 


fo Sb(x) 


foo si(x) 


= s 


t 0 
0 1 

t 0 
0 1 


t 0\ fl xt 1 
0 1/ VO 1 


- a 2 (t)fo(si(x )) 

- ai(t)foo ( st(x )) 


1 0 
0 t 


Si(xf ). 


ot 2 {x) 1 dx, 
a 2 (x)~ 1 dx. 


F x 


Let fo(x,t) and foo(x,t) denote the functions into brackets in the formulas above, respectively. 
We use formulas (3.1) and (3.2) above. For fo(x,t) we have the following: 

(1) if m ^ i + 1, then fo(x,t) = a 2 (t) — a 2 (t ) = 0; 

(2) if m = i, then fo(x,t) = —a 2 {t ); 

(3) if m i — 1, then fo(x, t ) = 0. 

For </>oo (x,t) we have: 

(1) if m ^ — i + 2, then fao(x,t) = 0; 

(2) if m = — i + 1, then f co (x,t) = ai(— tx~ 1 )a 2 (x); 

(3) if m —i, then f^(x,t) = a\(—tx~ 1 )a 2 (x) — ai(—tx~ 1 )a 2 (x) = 0. 

Therefore we have: 

Po(t) = — a 2 (t ) J a 2 ,(t l x)- 1 dx and Pao(t) = J cti(— t 1 ~ < ' 1 ~' l " > x~ 1 ) dx. 


This ends the proof of the lemma. 
We now have the following result. 


□ 


Theorem 3.f. The linear form ji can be extended to an B-invariant linear form on V if and 
only if one of the two conditions below is satisfied: 

(1) q =£ 1 in R and a\,a 2 are nontrivial. 

(2) q = 1 in R. 
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Proof. If ai,a 2 are ramified (that is, nontrivial on 0 X ), then: 




dx = 0. 


Thus fj. can be extended uniquely to an H-invariant linear form A on V, by setting A(/o) = 
A(/oo) = 0. If a ? ; is unramified for some i e {1, 2}, then: 


I 


a.i(x) 1 dx = q — 1. 


Fix a uniformizer vo of F and put Z{ = ai(w). 

(1) If i = 1, the condition on X(fco) writes: 

(3.3) (1 -zi)A(/oo) = z\(q- 1). 

If zi =£ 1, then (3.3) has a unique solution: 


A(/oo) =4-j 


g-i 


Zi 


If zi = 1, then (3.3) has a solution if and only if we have q = 1 in R, and in that case any value 
of A(/oo) in R is a solution. 

(2) If i = 2, the condition on A(/o) writes: 

(3.4) (1-z 2 )A(/ 0 ) = -z\-\q-l). 

If Z 2 / 1, then (3.4) has a unique solution: 

A(/o) - -4-‘ ■ P-f- 

L — Z2 

If Z 2 = 1, then (3.4) has a solution if and only if we have q = 1 in R, and in that case any value 
of A(/o) in R is a solution. 

This ends the proof of the theorem. □ 

Write d(V) for the dimension of V* H and e(V) for that of the subspace of H-invariant linear 
forms which are trivial on W. 

Theorem 3.5. - Let n denote the number of trivial characters among ai,« 2 - 

(1) If n = 0, then d(V) = 1 and e(V) = 0. 

(2) If n ^ 1 and q ¥= 1 in R, then d(V) = e(V) = n. 

(3) If n ^ 1 and q = 1 in R, then d(V) = n + 1 and e(V) = n. 

Proof. If q 1 in R, the result is as in the complex case. If q = 1 in R, then // can always 

be extended to an H-invariant linear form on V, that is, we have an exact sequence: 


0 (V/W) 


*H 


V 


*H 


w 


*H 


0 


of R-vector spaces and the dimension of W* H is 1. One easily checks that e(V) = n. The result 


follows. 


□ 
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3.2. The classification of GLi(F)-distinguished irreducible representations of GL 2 (F) 

For any irreducible (smooth) representation n of G, let d(ir) denote the dimension of its space 
of H-invariant linear forms. 

Recall that / denotes the quantum characteristic: 

f 0 if R has characteristic 0, 

[ the smallest positive integer k ^ 2 such that 1 + q + ■ ■ ■ + q k ~ 1 = 0 in R otherwise. 

An irreducible representation of G is said to be cuspidal if it does not embed in any V(ai, 02 ) 
with ai, «2 £ Gi. Just as in the complex case, we have the following result for cuspidal represen¬ 
tations. 

Proposition 3.6. All cuspidal irreducible representations n of G are H- distinguished, with 

d{ 7r) = 1. 

Proof. — See Paragraph 8.1 for a proof, where we treat the more general case of G n , n ^ 2. □ 

Now let St denote the Steinberg representation of G, that is the unique nondegenerate irre¬ 
ducible subquotient of V = Indg(l ® 1) (see [21, III. 1] ) . 

For the following lemma, see [13, §6]. 

Lemma 3.7. If f = 2, then St • y is cuspidal for all y e Gi. 

If / = 2, then Proposition 3.6 implies that St • y is H-distinguished with d( St • y) = 1 for all 
y £ Gi. Assume now that / A 2. Thus V has length 2 and we have an exact sequence: 

0-*y° det —► V • y = Indg(y ® y) ->• St • y -> 0 

of representations of G. If y is nontrivial, then any H-invariant linear form on V • y is trivial on 
y o det. We thus have d(St • y) = d(V ■ y) = 1. If y = 1, we have: 

d(St) ^ d(V) ^ d(St) + 1. 

As Ao and A® are H-invariant linear form on V which are nonzero on the subspace of constant 
functions, we get d{ St) = d(Y) — 1. Finally, we have the following result. 

Theorem 3.8. - (1) An irreducible representation of G is H- distinguished if and only if it 

is not a nontrivial 1-dimensional representation. 

(2) Let 7 r be an H- distinguished irreducible representation of G. Then d(ir) 2, with equality 
if and only if q = 1 in R and we are in one of the following cases: 

(a) 7 r is the Steinberg representation St and R has characteristic > 2 ; 

(b) it is a principal series representation V(l, y) = Indg(l®y) with y e Gi nontrivial. 


4. General results on modulo £ representations of G n 
4.1. More notation 

Let a = (rii,..., n r ) be a composition of n, that is, a family of positive integer whose sum is 
n. We denote by M a the subgroup of G n of invertible matrices which are diagonal by blocks of 
size ni,... ,n r respectively (it is isomorphic to G ni x • • • x G nr ) and by P Q the subgroup of G n 
generated by M Q and the upper triangular matrices. 
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We choose once and for all a square root of q in R. We write r a for the normalized Jacquet 
functor associated to (M a ,P„) and i a for its right adjoint functor, that is, normalized parabolic 
induction. If 7Ti,..., 7r r are smooth R-representations of G ni ,..., G Ur respectively, we write: 

(4.1) 7T1 X 7T 2 X • • • X TT r = i a ( 7Ti (g) 7T 2 ® ■ ■ • (g) 7T r ). 

Given a smooth representation 7r of finite length, we write [vr] for its semi-simplification and 
7T* for its contragredient. 

We write v for the normalized absolute value of F, giving value q~ l to any uniformizer. More 
generally, given integers k e Z and n ^ 1, we write: 

v k J 2 : 9 ~ (V?)“ fe ' val(det(9)) 

where is the square root of q in R that has been fixed above, val is the normalized valuation 
on F and det is the determinant map from G n to F x . 

We also write l n for the trivial character of G n , re ^ 1, and 1 for l^. 


4.2. The Geometric Lemma 


We give here a combinatorial version of Bernstein-Zelevinski’s Geometric Lemma [2] (see also 
[21, II.2.19]). Let a = (rei,... ,n r ) and (3 = (mi,... ,m s ) be two compositions of re ^ 1. For 
each i e {1,..., r}, let 7Tj e G ni . Let £$ a 'P be the set of all matrices B = (6jj) whose coefficients 
are non-negative integers such that: 


s r 

2 b h3 = n h i€{l,...,r}, Y b h3 = 3 6 {!> • ■ ■ 

j =1 i =1 

Fix B 6 and write a* = (b^i ,..., bi tS ) and j3j = (b\.j ,..., b r j) which are compositions of re, 
and rfij respectively. For all i e {1,... ,r}, the semi-simplification of r ai (-Ki) writes: 


km( 7r *)] = Y a i, 


(k) 

1 


UT 


(k) 
i,s 1 


r (fc) 

i,j 


(7'- / £ G hi 


Ti > 1. 


k =1 


For all j e {!,..., s} and all r-tuples k = (k \,..., k r ) with 1 ^ ki ^ r,;, we write: 


(k) (fci) (k r ) 

a j ~ a l,j X • • • X a r,j > 


which is a representation of G mj . Then we have: 

MTTl X ••• X 7T r )] = 


o 


(fc) 


) c 7 


(fc) 


B k 


in the Grothendieck group of finite length representations of M^. 


4.3. Cuspidal support 

An irreducible representation of G n with re ^ 1 is said to be cuspidal if it does not embed in 
any representation of the form (4.1) with r > 1. 

By [13, Theorem 2.1], for any irreducible representation ir e G n with re ^ 1, there are positive 
integers ni,... ,n r and cuspidal irreducible representations pi £ G ni with i e {1,..., r} such that 
re = rei + ■ • ■ + n r and n embeds in p\ x ■ ■ ■ x p r . Moreover, there is a permutation w of the set 
{ 1 , 2 ,..., r} such that it is a quotient of p w (\) x • • • x p w ( r y 

The family (p \,..., p r ), which depends on the choice of q , is unique up to permutation. Its 
class up to permutation, denoted [pi] + • • • + [p r ], is called the cuspidal support of it. 
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Proposition 4-1 ([13], Proposition 5.9). — Let ir,a be irreducible representations of 
G n ,G m respectively. Write [7Ti] + • • • + [7iy] and [cri] + • • • + [cr s ] for the cuspidal supports of 
7r and a respectively. Assume that for all i e {1,..., r}, j e {1,..., s} and k e Z, the cuspidal 
irreducible representations nt ■ v k and oj are not isomorphic. Then n x a is irreducible. 

4.4. Three lemmas about irreducibility 

The following lemma is a particular case of [12, Lemma 6.1], which will be of crucial impor¬ 
tance to us. Recall that e is the order (possibly infinite) of q in R x . 

Lemma 4-2. — Assume that e > 1. Let n ^ 2, and let p e G n _i, y e Gi. Then the represen¬ 
tation it = p x x possesses a unique irreducible quotient, denoted Q(7r) ; and a unique irreducible 
subrepresentation, denoted S(7r). 

Note that, by passing to the contragredient, we have: 

Q(p x x)* = S(p* x x' 1 ), s (p x x)* = Q(p* X X" 1 ). 

There is also a similar result for x x P, and we have: 

Q(x X p) = S (p x x), S(x X p) = Q(p x x). 

From this lemma we deduce the following example. 

Example 4-3■ — Assume that n ^ 2, and write: 

(4.2) V n = v][\ x ^ n+1 ^ 2 . 

If e > 1, Lemma 4.2 implies that V n has a unique irreducible quotient, denoted A n . We write: 

(4.3) A n = Q(V n ) = Q [y]l\ x ^ n+1 )/ 2 ), for e > 1. 

When e divides n, then A n is the trivial character (see Proposition 4.10). By taking the contra¬ 
gredient, A* is the unique irreducible subrepresentation of f n }/{ x i/~( n+1 )/ 2 . 

The following irreducibility criterion will also be very useful to us. 

Lemma 4-4 ([13], Lemme 2.5). — Let ir be a smooth representation of G n . Assume that 
there are two irreducible representations a e G a and r 6 Gj with a, b ^ 1 and a + b = n, such 
that: 

(1) tv is a subrepresentation of o x r and a quotient of r x a; 

(2) the multiplicity ofcr(g)T in r^ a ^(a x t) is 1. 

Then the representation n is irreducible. 

Finally, we will use the following lemma (which follows from [13, Proposition 2.2]). 

Lemma 4-5. Assume the induced representation (4.1) is irreducible. Then for all permuta¬ 
tion w of {1,2,... , n} there is an isomorphism ir w (i) x tt w ( 2 ) x • • • x tt w ^ ~ 7Ti x 7r2 x • • • x 7r r . 
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4.5. Classification of G n by multisegments 

In [13] Mfnguez and Secherre give a classification of the union of all G^’s in terms of multi¬ 
segments, that generalizes [26, 18, 22], We will need some properties of this classification, that 
we recall below. 

Given two half-integers a,b e ^Z, we write: 


a = b if 


a — b e eZ if R has positive characteristic, 
a = b otherwise. 


Definition J^.6. — (1) A segment is a pair (a, b ) of half-integers such that a ^ b. 

(2) Two segments (a, b ) and (c, d) are equivalent if 6 — a = d — c and a = c. The equivalence 
class of (a, b ) will be denoted [a, b] (and just [a] if b = a). 

(3) A multisegment is a formal finite sum of classes of segments, that is a element in the free 
semigroup generated by classes of segments. 


Let A = (Ai, A 2 , • • •) and fi = (/zi, / 12 , • • •) be two partitions of a given integer n. We say that 
A dominates //, denoted A !> //, if: 

Ai + • • • + Afc ^ /ii + ■ ■ ■ + Hk 

for all integers k ^ 1. We write A o /z if we have in addition A//i. 

Given a nonzero multisegment m = \a \, b\] + ■ ■ ■ + \a r , b r \, write n* = 6* — a* + 1 for all integer 
i e {1 ,..., r} and let A(m) denote the partition associated with (roi,ri 2 , • • • ,n r ). The length of 
m is the sum n = n\ + 77.2 + • • • + n r . 

One of the main results of [13] is the construction of a map m <—> Z(m) that associates to any 
multisegment m a class of irreducible representation Z(m) with the following properties: 

PI If m is a segment [a, b] of length n ^ 1, then Z([a, b]) is the character n^ +< ' n '^ 2 e G n . 

P2 If m = [ai, 61 ] + • • • + [a r , b r \, then Z(m) occurs as a subquotient of the representation 
Z([ai, 6 i]) x ••• x Z([a r , b r \) with multiplicity 1. 

P3 If 7T is an irreducible subquotient of Z([ai, b\]) x • • • x Z([a r , b r ]), then there exists a 
unique multisegment n = [ci,di] + • • • + [c s ,d s ] such that it = Z(n). Moreover, we 
have A(n) ^ A(m) and: 

j(N + ■■■ + M) = E(H + ••• + [!>.]). 

i —1 i=l 


P4 If £: is a half-integer, then Z([ai + k,b\ + k\ + ■ ■ ■ + \a r + k, b r + A:]) = Z(m) • u k . 

P5 The contragredient of Z(m) is Z(m*) with m* = [—b\, —ai] + ■ ■ ■ + [— b r , — a r ]. 

We finally have the following definition and result. 

Definition 4-7. Two segments [a, b] and [c,d\ are linked if: 

(1) the length of [a, b] is greater than or equal to that of [c, d], and there exists a half-integer 
k such that c ^ k ^ d, and either k = b + loik = a— 1; 

(2) the length of [c, d] is greater than or equal to that of [a, b ], and there exists a half-integer 
k such that a ^ k ^ b, and either k = d+ l or k = c— 1. 
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Proposition 4-8 ([13], Theoreme 7.26). Let Ai,..., A r be segments. The representation 
Z(Ai) x • • • x Z(A r ) is irreducible if and only if for all i + j, the segments A j, A j are not linked. 


4.6. Product of two characters 

Here are some useful properties of the representation Z([a,b]) for a segment [a, b]. 

Proposition 4-9. Let [a, b] be a segment of length n ^ 2, and let k e {1 ,... ,n — 1}. 

(1) We have r^, n -k)(Z([a, b ])) = Z([a, a + k — 1]) ® Z([a + k, b ]). 

(2) ITe 6ave n _ fc )(Z([a, 6])) = Z([a +n —6])(x)Z([a, a + n ——1]) where f^ k n _ k ) denotes 
the Jacquet functor associated to M( fen _ fc ) and the parabolic subgroup opposite to P ( k ,n-k)- 

(3) Assume e > 1. T/ien Z([a, 6 — 1]) x u b has a unique irreducible subrepresentation and 
Z([a + 1,6]) x u a has a unique irreducible quotient, both isomorphic to Z([a,6]). 

Proof. — See [13, Propositions 7.16 and 7.17]. □ 

Proposition 4-10. Assume e > 1. Let a =5 b be integers and write ir(a,b) = Z([a,6]) x 1. 

(1) If a ^ 1 and 6^—1, then ir(a,b) is irreducible. 

(2) If a = 1 and 6^—1, then n(a,b) has length 2 and we have an exact sequence: 

0 —> Z([a, 6] + [0]) —> 7r(a, 6) —* Z([a — 1,6]) —* 0. 

(3) If a ^ 1 and b = — l, then n (a, 6) has length 2 and we have an exact sequence: 

0 —> Z([a, 6 + 1]) —> 7r(a, 6) —> Z([a, 6] + [0]) —> 0. 

(4) If a = 1 and b = —1, f/ien 7r(a, 6) /ias length 3 with irreducible subquotients Z([a — 1,6]), 
Z([a, 6 + 1]) and Z([a, 6] + [0]). 

Proof. — Case 1 follows from Proposition 4.8. Moreover, the representation Z([a, 6] + [0]) always 
occurs as a subquotient with multiplicity 1 and the other irreducible subquotients of 7r(a, 6) are 
of the form Z(n) with A(n) o(n — 1,1), where n = 6 —a + 2. Therefore we have A(n) = (n), which 
implies that n is a segment. Moreover, such an n must be of the form [a, 6+1] with 6 = — 1 or 
[a — 1,6] with a = 1. 

Assume that a ^ 1 and 6 = —1. By the geometric lemma, the Jacquet module r(„_i > i)(7r(a, 6)) 
is made of the subquotients Z([a, 6]) ® 1 and tt( a, 6 — 1) ® n 6 , and both are irreducible. Thus the 
representation 7r(a, 6) has length 2. But Proposition 4.9 shows that Z([a, 6 + 1]) occurs as a 
subrepresentation of 7r(a, 6). The result follows. 

The case where a = 1 and 6 ^ —1 is treated in a similar way. Thus it remains to study the 
case where a = 1 and 6 = —1. In this case, 7r(a, 6—1) has length 2, thus 7r(a, 6) has length ^ 3. 
By Proposition 4.9 we see that the length is actually 3 and we get the expected result. □ 


n 


Example 4-H- 

(4.4) 

Assume e > 1 and look at Example 4.3 for the definition of A n . Then: 


Assume that n ^ 2 and write: 

n„ = z' 


(4.5) 


A n = 


n„ if e does not divide n, 


lr 


if e divides n. 
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We also get: 


A 


* 

n 


II* = Z([—^ | + [—if e does not divide n, 
l n if e divides n. 


If we want to go further, we need more properties of the representation Z(m) for a multisegment 
m. Given xi > X '2 e Gi, we write St(xi,X2) for the unique nondegenerate irreducible subquotient 
°f Xi x X2 (see [13, §8]). If St 2 is the Steinberg representation of G 2 as in Paragraph 3.2, then: 

[ X1 x X2 if Xi x X2 is irreducible, 

St(xi,X 2 ) = j St 2 - Xi^ 1/2 if X2 = X\v, 

{ st 2 • X2U 1/2 if XI = Xiv- 

Note that we have St(x2iXi) = St(xi,X 2 )- The following proposition follows from [12, §3.3.2], 


Proposition f,.12. — Let m be a multisegment of length n and of the form [a, b] + [c, d]. Assu¬ 
me that b — a ^ d — c. Write k = d — c + 1 and: 

(i(m) = (1,..., 1 ,2,... , 2) with 1 occurring n — 2k times and 2 occurring k times, 

St(m) = u a ®---®u a+n_2fc_1 ®St(u a+n_2fc ,z/ : )®---®St(zAz/). 

Then Z(m) has the following property: 

P 6 Z(m) is the unique irreducible subquotient o/Z([a, 6]) x Z([c, d]) whose Jacquet module 
with respect to r^( OT ) contains St(m) as a subquotient. 

Proposition f.13. — Let a,b e Z with a ^ b and write n (a, b ) = Z([a, b ]) x Z([0,1]). Assume 
that e > 1 . 

(1) Z([a, b] + [0,1]) occurs as a subquotient ofir(a,b) with multiplicity 1. 

(2) 7/6 = 0, then Z([a, b + 1] + [0]) occurs as a subquotient of n(a,b) with multiplicity 1. 

(3) If a = 1, then Z([a — 1,6] + [1]) occurs as a subquotient ofir(a,b) with multiplicity 1. 

(4) If b = —1, then Z([a, 6 + 2]) occurs as a subquotient of 7 r(a, 6). 

(5) If a = 2, then Z([a — 2, 6]) occurs as a subquotient of n (a, 6). 

(6) 7/6 = 0 and a = 1, t6en Z([a — 1,6+ 1]) occurs as a subquotient of 7r(a, 6). 

Any irreducible subquotient of n (a, 6) is one of the representations in Cases 1 to 6. Moreover, 
if e> 2, the multiplicities in Cases f, 5 and 6 are equal to 1. 


Proof. — Case 1 follows from P2. Write n = 6 — a + 3. The other irreducible subquotients of 
it (a, b ) are of the form Z(n) with A(n) o(n — 2, 2). Thus we have A(n) = (n — 1,1) or A(n) = (n). 
If 6 = 0 (resp. a = 1) then Z([a, 6 + 1]) x 1 (resp. Z([a — 1, 6]) x v) is a subquotient of 7 r(a, 6) by 
[13, Lemme 7.34]. It follows from Proposition 4.10 that the representations in Cases 2, 3 and 
6 occur in 7 r(a, 6). Cases 4 and 5 are treated similarly. We now show that these are the only 
possible subquotients of 7 r(a, 6) and they appear with the specified multiplicity. 

Assume first that n = [c, d] + \h ] with d — c+l = n — 1. Then: 

M-(n) = (1,... ,1,2), 

St(n) = v c ® u c+1 ® • • • ® v d ~ l ® St(u d , u h ). 

By using the geometric lemma, the semi-simplification of T( n _ 2) 2 )('r(a, 6)) is equal to: 

Z([a, 6]) ® Z([0,1]) + (Z([a, 6 - 1]) x 1) ® (u 6 x v) + 7r(a, 6 — 2) ® Z([6 - 1, 6]). 
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If Z(n) occurs as a subquotient of 7r(a, 6), then St(n) occurs in *V(n)(( z ([>,&-!]) x x v)) 

and St(u d , v h ) occurs in u b x u with multiplicity 1, which implies that St(u d , v h ) = St(re 6 , re) and 
that Z(n) occurs in n(a,b) with multiplicity 1. By the geometric lemma, we get: 

n —3 

r (1 ,..., 1) (Z([a, b - 1]) x 1) = ® ® v a+k ~ l ® 1 ® re a+fc ® • • • ® re 6-1 . 

k =o 

Thus there is a k e {0,..., re} such that: 

re c ® re c+1 (x) • • • ® v d ~ l = re“ ® • • • (x) re a+fc-1 ®1® v a+k ® • • • ® i/ 6-1 . 

Since e > 1, comparing the exponents in the left hand side and the right hand side shows that 
k must be either 0 or re — 3 = b — a. If k = (1, then: 

re c ®re c+ 1 ®---®re d_1 = l®re a ® 

Thus we have c = 0, a = 1, d = b and h = 1. It follows that n = [a — 1, 6] + [1]. If k = n — 3, 
then: 

v c ® v c+1 ® • • • ® i/ d_1 = v a ® • • • ® v b ~ l ® 1. 

Thus we have c = a, 6 = 0, d = 1 and h = 0. It follows that n = [a, b + 1] + [0]. 

Assume now that n = [c, d ] is a segment. Thus: 

p(n) = (1, - - -, 1), 

St(n) = ® is c+1 ® • • • ® v d . 

By using the geometric lemma, we get: 

r n(„)( 7r ( a ; b)) = v a ® • • • ® v a+r ~ l ® 1 ® v a+r ® • • • ® i/ a+s_1 ® v ® i/ a+s ® • • • ® v b . 

If Z(n) occurs as a subquotient of 7r(a, 6), there are integers r ^ s in {0,..., n) such that: 

v c ® ■ ■ • ® v d = v a ® • ■ ■ ® i/ a+r_1 ® 1 ® v a+r ® • • ■ ® z/ a+s_1 ® v ® jy a+s ® ■ • • ® v b . 

If e > 2, comparing the exponents in the left hand side and the right hand side shows that the 
only possible values for r, s are: 

(1) r = s = 0 (thus a = 2); 

(2) r = s = n (thus b = —1); 

(3) r = 0 and s = n (thus a = 1 and b = 0). 

In all these cases, St(n) occurs with multiplicity 1. 

If e = 2, there are more possible values for r, s (the condition is that s — r is even) and St(n) 
may occur with multiplicity greater than 1. □ 

4.7. Derivatives 

By [21, III. 1], there is a theory of derivatives for mod i representations of G n , n ^ 1 just as in 
the complex case. Given a smooth representation 7r of G n , n ^ 1 and an integer k e {0,..., n}, 
we will write ir^ for its k th derivative, which is a smooth representation of G n _fc (where Go 
stands for the trivial group in the case k = n .) 

The k th derivative functor is exact from the category of smooth ^-modular representations of 
G n to that of smooth Gmodular representations of G n _fc, for all k e {0,... , re}. It is compatible 
with twisting by a character, that is, we have (7r • x)^ = ' X f° r an y representation ir of G n , 

any character x 6 Gi and any k e {0,... , re}. 
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Recall that [ 77 ] denotes the semi-simplification of a finite length representation 7 r. 

Lemma 4-14- ~ (1) Given a cuspidal irreducible representation p of G n , its k th derivative 

is zero for all k e {1,..., n — 1}, and we have p^ = 1 for k = n. 

(2) Given a segment [a, b], the first derivative 0 / Z([a, b ]) is Z([a, 6—1]), and its k th derivative 
is zero for all k e {2,..., n}. 

(3) Let ir, cr be finite length representations of G n ,G m respectively, with m ^ n ^ 1. Then : 

[(tt x a) (fc) ] = [tt x a (fe) ] + [tt (1) x + ■■■ + [vr (i) x a^] 

for all k e {0,..., n + m}, where i = min(n, k ). 

Proof. — Points (1) and (2) follows from V.9.1 (a) and (b) in [22], For (3), see [21, III.1.10]. □ 

5. On the e = 1 case 

In this section, we assume that e = 1 and n ^ 2. Write K n = GL n (0) and let K n (l) be the 
normal subgroup of K n made of all matrices that are congruent to 1 mod p. Both are compact 
open subgroups of G n , and the quotient K„/K„(l) is canonically isomorphic to the finite group 
GL n (g) of n x n invertible matrices with entries in the residue field of 0. 

Given a smooth representation (7r, W) of G n , write W for the space of K n (l)-fixed vectors of 
W and write W for the representation of GL n (g) on W. 

This defines an exact functor from the category of smooth R-representations of G n to that of 
R-representations of GL n ,(g). 

We have defined two representations V n and II n in (4.2) and (4.4). Note that \ n = C“ (X, R) 
with X = P(„_i i i)\G n . Its contains II n as a subquotient with multiplicity one, l n with some 
multiplicity and no other irreducible subquotient. It is a selfdual representation of G n . 

Thanks to the Iwasawa decomposition G n = P( n _ 11 )K n , the restriction of V n to K n is W n = 
C“(Y, R) with Y = (K n n P(n-i,i))\K n - Therefore we have: 

V n = 6*(Y/K n (l),R), 

which identifies with the space of R-valued functions on X = P(n-i,i)(<?)\GL n (q), where we write 
P(n— 1 , 1 )(<?) for the standard maximal parabolic subgroup of GL n (q) corresponding to (n — 1,1). 

Lemma 5.1. Forn ^ 2, there exists a unique irreducible representation ir n ofGL n (q) having 
the following properties: 

(1) If £ does not divide n, then \ n is semisimple of length 2, with irreducible subquotients \ n 
and 7 r n . 

(2) If £ divides n, then V n is indecomposable of length 3, with irreducible subquotients l n 
(with multiplicity 2 ) and 7r n . 

Proof. — Note that l n occurs as a subrepresentation of V n (the space of R-valued constant func¬ 
tions on X). Write if for the GL n (g)-invariant linear form on V n that associates to a function 
the sum of its values on X. The set X has cardinality: 

n n _ 1 

(GL n (g) : P(„_i ) i)(g)) = - _ - = 1 + q +■■■ + q n 1 

which is 0 in R if and only if £ divides n. Thus the constant functions belong to the kernel of if 
if and only if £ divides n. According to [8, 11], we have the following properties: 
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(1) The kernel S n of if (denoted S(n_ 1>:L ) in [8], whereas V n is denoted M( n _i.i)) has a unique 
irreducible quotient Ti n . 

(2) The semi-simplification of V n contains 7r n with multiplicity 1 and l n with some multiplicity 
^ 1, and no other irreducible subquotient. 

By [8, 20.7], the multiplicity of I n in V n is 1 if £ does not divide n, and 2 otherwise. It remains 
to prove that V n has the expected structure. 

We first assume that £ does not divide n. Since l n occurs as a subrepresentation of V n , n n 
must be a quotient of V n . Since V n is selfdual, it follows that 7r n is selfdual, thus it also occurs 
as a subrepresentation of V n . We thus have two nonzero maps 7r n —» V n and V n —> 7r n , whose 
composite is nonzero (or else it would contradict the fact that ir n occurs with multiplicity 1). 
Therefore V n is semisimple. 

Assume now that £ divides n. By [8], the representation S n is indecomposable (it has length 2 
and a unique irreducible quotient). Since V n is selfdual, it implies that V n is indecomposable. □ 

Proposition 5.2. — (1) The representation II n is irreducible and isomorphic to ir n . 

(2) If £ does not divide n, the representation V n is semisimple of length 2. 

(3) If £ divides n, the representation V n is indecomposable of length 3, with irreducible sub¬ 
quotients l n (with multiplicity 2 ) and Ii n . 

Proof. By [21], II.5.8 and II.5.12, all irreducible subquotients of V n have level 0, thus they 
are not killed by the functor n *—> T. 

We first assume that £ does not divide n. By Lemma 5.1, the representation V„ has length 
2, with irreducible subquotients II n and l n , thus Lf n must be irreducible and isomorphic to 7r n . 
The same argument as in the proof of Lemma 5.1 shows that V n is semisimple. 

Assume now that £ divides n. By Lemma 5.1 the representation V n has length ^ 3. Assume 
it has length 2. Then the argument of the proof of Lemma 5.1 implies that \ n = l n ®II n . Thus 
the one-dimensional space HomG n (V n , l n ) is generated by a linear form A which is nonzero on 
the subspace of constant functions. Since K n (l) is a pro-p-group, K n (l(-invariant and K n (l)-co- 
invariant vectors of V n are canonically identified. The K n -invariant linear form A thus induces a 
GL n ((?)-invariant linear form on V n , which is equal to upto a nonzero scalar. But tjj is zero on 
constant functions, which contradicts the fact that A is nonzero. This gives us a contradiction, 
and thus V n has length 3. Now since V n is indecomposable, it follows that V n is indecomposable. 
We also get that II n must be irreducible and isomorphic to 7r n . □ 


Definition 5.3. 


Assume e = 1 and let n ^ 2. In parallel with Example 4.11, we define: 


A 


n 


n n if £ does not divide n, 
l n if £ divides n. 


In conclusion, if we summarize Example 4.11 and Definition 5.3, we get the following definition 
of A n . 


Definition 5-4- — Assume e is arbitrary, and recall that / is the quantum characteristic (see 
Paragraph 3.2). For n ^ 2, we define: 


(5.1) 


A, 


n n = Z( [— ~2^ , + t 1 ^]) if / does not divide n, 

l n if / divides n. 
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Thanks to Example 4.11, note that we also have: 


(5.2) 


A* 


II* = Z([— ~ 2 ~i I + [— *^]) if / does not divide n, 

l n if / divides n. 


If we look at Proposition 5.2, we also have the following property (for arbitrary e ^ 1). 


Remark 5.5. - For n ^ 2, if / does not divide n, then A n is an irreducible quotient of V n . 


6. Computing the derivatives of A„ and II„ 


In this section, we assume that e is arbitrary. Remind (see (4.2), (4.4) and (5.1)) that we have 
defined representations V n , H n and A n for all n ^ 2. By Propositions 4.10 and 5.2, we have: 


( 6 . 1 ) 


[Vn] 


n n + v n if / does not divide n, 

II n + v n + l n if / divides n, 


in the Grothendieck group of finite length representations of G n . Let us compute the derivatives 

of n„. 


Lemma 6.1. — Suppose that n ^ 2. 

(1) If f = n = 2, the derivative II^ is zero. 

(2) Otherwise we have: 



In—2 X Z> +1 )/ 2 

K-i • ^ 1/2 


if f does not divide n, 
if f divides n. 


(3) We have IIn 1 = l „,-2 and H'rP is zero for all k ^ 3. 


Proof. — By Leibniz’s rule (see Lemma 4.14(3)), we have: 

[Vi 1 '] - [l„-2 X .AW/2] + 

in the Grothendieck group of finite length representations of G n -i- Since the k th derivative of a 
character is zero for k ^ 2, we have Vn ' = l n -2 and Vn J is zero for all k 5= 3. The k derivative 
functors being exact, the expected result follows from (6.1) together with Propositions 4.10 and 
5.2. □ 


Corollary 6.2. — Suppose that n ^ 2. 

(1) We have: 

if f does not divide n, 
if f divides n. 


AW = 


l n -2 X l / n+1 )/2 
- 1/2 
v n-\ 


(2) The second derivative A„ ; is equal to l „-2 if f does not divide n, and is zero otherwise. 

(3) The k th derivative A^ is zero for all k ^ 3. 

Remark 6.3. — Since 11* = n„ • by Properties P4 and P5, we get the derivatives of Lt* 
and A* from Lemma 6.1 and Corollary 6.2. 


Example 6 . 4 ■ — (1) We have St 2 = Z([—1/2] + [1/2]) = II 2 • a 1 . If / = 2, the representa¬ 

tion St 2 is cuspidal thus its first derivative is zero. Otherwise, we have (St 2 )*' 1 ^ = v 1 ! 2 . 
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(2) Let St 3 denote the nondegenerate irreducible subquotient of v 1 x 1 x u, that is: 

St,3 = Z([—1] + [0] + [1]) 

(see [13, §8]). If / = 3, then St 3 is cuspidal ([13, §6]) thus its first and second derivatives are 
zero. If / + 3, then: 

[W 1 x 1 x v\ = 1 3 + A 3 • v~ l + (A 3 )* • v + St 3 

in the Grothendieck group of finite length representations of G 3 . We thus get (St 3 )^ = St 2 • v 1 / 2 
and (St 3 )^ 2) = v. 


7. A modular version of Badulescu-Lapid-Mfnguez’s juxtaposition criterion 

In Paragraph 4.5 we have defined Z(A) for A a segment. In [13] an irreducible representation 
L(A) is also introduced. We will need it only for segments of length ^ 2. 


Definition 7.1. Let a be a half-integer. Then L([aj) = Z([aj) = v a and: 


L([a, a + 1]) 


Q(/x/ +1 ) if e > 1, 
A 2 • v a+1 / 2 if e = 1. 


Remark 7.2. — Note that we have f(i,i)(L([a, a + 1])) = v a+l ® v a for all a e |Z. 


If we write St 2 for the Steinberg representation of G 2 as in Paragraph 3.2, then we have: 


L([o, a + 1]) 


St 2 • i/ a+1 / 2 if / ^ 2, 
u a ~l /2 if / = 2. 


Note that A 2 = St 2 • u if f 7= 2. 


Lemma 7.3 ([13], Theoreme 7.26). - Given two segments A and A' of length 2, the 

representation L(A) x L(A') is irreducible if and only if A and A' are not linked. 

Following [3, Definition 2.1], say that two segments [a, b] and [c, d] are juxtaposed if we have 
c = 6+ lora = d-l-l. 


Proposition 7.f. Let A, A' be two segments. Assume A' has length 2. Then Z(A) x L(A') 
is reducible if and only if A and A' are juxtaposed. 

Proof. — By twisting by a character, we may and will assume that A' = [0,1]. If e = 2, then 
L([0,1]) is cuspidal (see Lemma 3.7), thus the result is true by Proposition 4.1. If A and [0,1] 
are juxtaposed, Zelevinski’s argument (see [26, §2]) proves that it is reducible. In particular, if 
e = 1, then A and [0,1] are always juxtaposed. We are thus reduced to prove the following. 

Proposition 7.5. Assume that e > 2 and let a ^ b be integers. Then Z([a, b]) x L([0,1]) is 
reducible if and only if e divides 6+1 or a — 2. 

We write n = Z([a,6]) x L([0,1]). When e divides 6+ 1 or a — 2, Zelevinski’s argument proves 
that 7r is reducible. 

We now assume e does not divide 6 + 1 nor a — 2, and write = for the relation of congruence 
mod e in Z. We thus have a ^ 2 and 6^—1. The proof is by induction on n = 6 — a + 1. 

If n = 1 then tt = u a x L([0,1]) and the result follows from Lemma 7.3 since the segments [a] 
and [0,1] are not linked. 
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Assume now n ^ 2. Our goal is to find irreducible representations cr, r, of degree u, v res¬ 
pectively, such that 7r occurs as a subrepresentation of jxt and as a quotient of r x cr, and such 
that cr®r occurs with multiplicity 1 in r^ u ^(a x r). We will distinguish the following cases: 

(1) a# -1,1 

(2) a = —1 

(3) a = 1 and 6^0,2 

(4) a = 1 and 6 = 0,2 and e > 3 

(5) a = 1 and 6 = 0 and e = 3 

In Case 1, since a ^ 1 and thanks to the inductive hypothesis, 7r embeds in: 

(7.1) u a x Z([a + 1, 6]) x L([0,1]) ~ u a x L([0,1]) x Z([a + 1,6]) 

and u a x L([0,1]) is irreducible because a ^ — 1. Since Z([a, 6]) is a quotient of Z([a + 1,6]) x u a , 
we can choose a = v a x L([0,1]) and t = Z([a + 1,6]). We compute the multiplicity of cr® r in 
r ( 3 ,n-i)( CJ x T ) by applying the geometric lemma. For this multiplicity to be 1, it is enough to 
prove that cr does not occur as a subquotient of the following representations: 

(1.1) v a x 1 x z/ a+1 ; 

(1.2) L([0,1]) x u a+1 ; 

(1.3) v a x Z([o + l,o + 2]); 

(1.4) v x Z([a + 1,a + 2]); 

(1.5) Z([a + 1,a + 3]). 

This follows from [13, Theoreme 8.16]. 

In Case 2, Equation (7.1) in addition with the fact that L([0,1]) embeds in v x 1 implies that 
7T is a subrepresentation of: 

v~ l xi/xlx Z([a + 1, 6]). 

But 7r is also a quotient of: 

Z([o, 6]) x 1 x v ~ 1 x Z([a, 6]) x v 

which itself is a quotient of the representation 1 x Z([a + 1,6]) x u -1 x v. We thus can choose 
cr = u -1 x v and r = 1 x Z([a + 1,6]). Again, by the geometric lemma, it is enough to prove 
that (7 does not occur as a subquotient of u” 1 x 1, v x 1, Z([0,1]) or lxl, which follows easily. 

In Case 3, we embed Z([a, 6]) into Z([a, 6 — 1]) x u b and show by a similar argument that we 
can choose cr = Z([l,6 — 1]) x L([0,1]) and r = u b . By using the geometric lemma, it is enough 
to prove that u b is different from 1 and which is immediate. 

In Case 4, we prove the following more general lemma. 

Lemma 7.6. Assume e > 3. Then Z([l, 6]) x L([0,1]) is irreducible for any 6 ^ 1, 6 ^ —1. 

Proof. — We first treat the case where 6 = 2 (the case where 6=1 has already been done). We 
embed n = Z([l,2]) x L([0,1]) in: 

Z([l,2]) x v x 1 ~ v x Z([l,2]) xl-=>uxuxz/ 2 xl 

and we choose a = v x v and t = v 2 x 1. 

Now assume 6^3. We embed Z([l,6]) in Z([l,2]) x Z([3,6]) and then choose cr = Z([l,2]) 
and r = Z([3, 6]) x L([0,1]). By the geometric lemma, it is enough to prove a does not occur in: 

(4.1) v x u 3 ; 

(4.2) v x v\ 
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(4.3) L([0,1]); 

(4.4) z / 3 x z/; 

(4.5) Z([3,4]). 

This is immediate. □ 

In Case 5, n is of the form 3 k for some k ^ 1, and we write flfc = Z([l, 3k]). 

Lemma 7.7. - The representation fli x L([0,1]) is irreducible. 

Proof. — Let £ be an irreducible subquotient of 7 r = Qi x L([0,1]). It is thus a subquotient of 
the representation Z([l,3]) x v x 1 . By using Properties P2 and P3, we deduce that £ is of the 
form Z(m) where m is a multisegment in the following list: 

(5.1) m= [0,4]; 

(5.2) m = [0,3] + [1]; 

(5.3) m = [1,4] + [0]; 

(5.4) m= [0,2] + [3,4]; 

(5.5) m= [2,4]+ [0,1]; 

(5.6) m= [1,3] + [0,1]; 

(5.7) m = [1,3] + [0] + [1]. 

We will prove that Case 5.7 is the only possible case, which implies that fli x L([0,1]) is irredu¬ 
cible and equal to Z([l,3] + [0] + [1]). By the geometric lemma, we get: 

[ r ( 3 , 2 )( 7r )] = Z([l, 3]) ® L( [0,1]) + (Z([l, 2 ]) xi/)® (lx l) + (i/xL([0,l]))®Z([2,3]) 

and each of these three subquotients is irreducible. Since r ( 3 2 )(Z([0,4])) = Z([0, 2]) ® Z([3,4]), 
we see that Z([0,4]) cannot occur as a subquotient of 7 r. 

Now the semi-simplification of C 1 , 2 , 2 ) C 73- ) is equal to: 

v ® Z([2,3]) ® L([0,1 ]) + 1 / ® L([0,1]) ® Z([2, 3])+r0(rxl)0 Z([2, 3]) 

+ v ® Z([l, 2])®(lxl) + !/0 ( v 2 xi/) 0 (lx 1 ). 

By using Proposition 4.12, we see that Cases 5.4, 5.5 and 5.6 cannot occur. 

Now the semi-simplification of ?"(i i i > i i 2 )( 7r ) is equal to: 

V ® 1 (X) V ® Z([2, 3]) + v ® z / 2 ® 1 (X) L([0,1]) + v ® z / 2 ® v ® (1 ® 1) 

+ 2- (z/ ® z/ ® 1 ® Z([2, 3])) + 2 • (z/ ® v ® z / 2 ® (1 x 1 )). 

By using Proposition 4.12, we see that Case 5.2 cannot occur. 

It remains to treat Case 5.3. The semi-simplification of r( 113 )(Z([l, 4]) x 1 ) is equal to: 

2 /® z / 2 ® (Z([0,1]) x 1 ) + 1 ® z /0 Z([2,4]) + 2 / 01 ® Z([2,4]). 

By Proposition 4.10(2) and the geometric lemma, we get: 

[ r { 1 , 1 > 3 ) (Z([l,4]) + [0])] = 2 /® 2/ 2 ® (Z([0,1]) x 1) + 2 / ® 1 ® Z([2,4]). 

On the other hand, the semisimplification of ^( 1 , 1 , 3 ) (tt) is equal to: 

2 / 01 ® Z([l, 3]) + 2 • ( 2 / ® 2 / ® (Z([2, 3]) x 1))+ 2 /® 2 / 2 ®(l x L([0,1])) 

and each of the individual subquotients is irreducible. Therefore, Case 5.3 cannot occur. □ 
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The proof is now by induction on k. We embed fifc+i i n t° ^1 x ^k and choose a = Qj and 
r = L([0,1]) x By using the geometric lemma, we have to prove that, for all 0 ^ i ^ 2, the 
factor (7 0 r does not occur as a subquotient of any of these three representations: 

(5.A) Z([l,i]) x L([0,1]) x Z([l,l -i])<g>Z([i + l,3]) x Z([2 - i,3fc]); 

(5.B) Z([l, *]) x v x Z([l, 2 — z]) ® Z([i + 1,3]) x 1 x Z([3 — i, 3/cJ); 

(5.C) Z([l,i]) x Z([l,3-i])®Z([i + 1,3]) x L([0,1]) x Z([4 - i, 3fc]). 

This follows by using Property P3. (Notice that the term (5.A) does not appear if * = 2). 

This ends the proof of Proposition 7.5. □ 


8. Distinguished representations 

For n ^ 2, we write H n for the subgroup of G n made of all matrices of the form: 



Definition 8.1. Assume that n ^ 2. A smooth R-representation (7r, V) of G n is said to be 
H n -distinguished if V possesses a nonzero H n -invariant linear form. 

If the space HoniH n (V, R) has finite dimension over R, we denote this dimension by d(n). 

8.1. Cuspidal representations 

Just as in the complex case (see [15]), we have the following result. 

Theorem 8.2. — Let n ^ 2 and let p e G n be a cuspidal representation. Then p is distinguished 
if and only if n = 2. When it is the case, we have d{p) = 1. 

Proof. — Write P n for the mirabolic subgroup of G n , that is the subgroup made of all matrices 
with last row (0,... ,0,1). By [21, III,Theorem 1.1], the restriction of p to P n is isomorphic, 
just as in the complex case, to the compact R-induction: 

(V’n) 

of a generic character if n of the standard maximal unipotent subgroup U n of G n . As P n = H n U n , 
the restriction of p to H n is isomorphic to the compact R-induction indy" nU which carries 

a nonzero H„-fixed R-linear form if and only if if n is trivial on H n n U n . This happens if and 
only if n = 2, in which case we have d(p) = dimHomH 2n u 2 (^2,1) = 1- □ 

8.2. Distinction and contragredient 

We have the very useful following result. Assume n ^ 2. 

Proposition 8.3. — Assume that the characteristic of R is not 2, and let it e G n . Then n is 
H n - distinguished if and only n* is. 

Proof. In the complex case, this result follows from the well-known fact (due to Gelfand and 
Kazhdan) that the contragradient of an irreducible representation n e G n is isomorphic to ir o a 
where a is the involution: 

g >—> transpose of g -1 

of G n . For a field R of characteristic not 2, this argument still holds (see [13, Remarque 2.7]). □ 
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Remark 8-4- — Note that the condition e > 1 implies that the characteristic of R is not 2. 

In the case where R has characteristic 2, it is likely that the Gelfand-Kazhdan property still 
holds (thus so Proposition 8.3 does) but we won’t need it in this case. 

Proposition 8.5. - Assume that the characteristic of R is not 2 and n = n± + n 2 where n±,n 2 

are positive integers. Let 7r, e G ni for i = 1,2. Then tt\ x tt 2 is H n - distinguished if and only if 
7r| x 7T* is. 

Proof. In the complex case, the proof follows from the above-mentioned theorem of Gelfand- 
Kazhdan, the existence of a group automorphism of G n which maps P(m,n 2 ) to P( n2 ni ), tt\ x 712 
to 7r| x tt* and G n _i to a conjugate of G n _i. All these are valid for R (see [21, III.1.16] and 
[26, 1.9 ]) if it has characteristic not 2. Hence: 

Hom Gn _ 1 (7Ti x tt 2 ,R) ^ Hom Gn l (7r| x tt*,R) 

and our claim follows. □ 

8.3. The Bernstein-Zelevinski filtration 

For i e {0,1,..., n}, we write R* jn for the subgroup of matrices of G n of the form: 



such that g e Gj and h is an upper triangular and unipotent matrix of G n _j. In particular, Ro,n 
is the standard maximal unipotent subgroup U n of G n and R n _i in is the mirabolic subgroup P n 
of G n . Fix a nontrivial smooth character if : F —> R x and, for i e {0,1,... , n — 1}, write for 
the generic character of Uj defined by: 

ifi(h) = if(hi t 2 H-h 

for all h e Uj. From [21, III.1.3], we have the following result. 

Theorem 8.6. - Let V he a representation of G n . There are P n -stable subspaces Vo,..., V n 
of V such that {0} = Vo Q Vi Q ■■■ c V n = V and: 

Vi+i/Vi ^ indt(V M ^ /2 ®^-i) 

for all i e { 0 ,..., n — 1 }. 

As in the complex case (see page 54 of [5] and [15, Proposition 1]), we get the following result 
by using the Bernstein-Zelevinski filtration. 

Lemma 8.7. Let n be a smooth representation of G n with n ^ 3, and assume that: 

(1) 7rW does not have any quotient isomorphic to v n ^i ; 

(2) 7r^ 2 ^ does not have any quotient isomorphic to l n - 2 - 
Then n is not distinguished. 


MODULAR REPRESENTATIONS OF GL(n) DISTINGUISHED BY GL(n - 1) 


25 


8.4. The Three Orbits Lemma 

As in the complex case [19], we have the following very useful lemma. 

Lemma 8.8. Let n ^ 2 and k e {1,... , n — 1} be integers, and let p e G k and t e G n - k . 
Assumme p x r is H n - distinguished. Then at least one of the following conditions is satisfied: 

(A) p = 2 k ^ 2 and r • v k / 2 is R n - k -distinguished. 

(B) p • is H k -distinguished and t = 2 ^ 2 . 

(C) p^ ■ zy - ( n-1-fc )/ 2 and t*^ ■ have a trivial quotient. 

Conversely, if p e and r e G n _ k satisfy (A) or (B), then p x r is H„- distinguished. 

Proof. - The proof in just as in the complex case (see [19, Section 5]). □ 

Remark 8.9. — Notice that if r is smooth (not necessarily irreducible), we still have conditions 
similar to Lemma 8.8. We will have the occasion to use this in the case where: 

P = v n - 3 > r = St 2 • u _(n_2)/2 x x, X e Gi. 

In this case, p x r is H n -distinguished if and only if r • i/( n-3 )/ 2 is H 3 -distinguished. 

Corollary 8.10. - Let n ^ 3, and let n e G n be H n - distinguished. Then one of the following 

properties holds: 

(1) 7r = l n _2 x t for some irreducible cuspidal representation re G 2 . 

(2) The cuspidal support of it is made of characters of Gi. 

Proof. There are irreducible cuspidal representations n,... ,T r such that 7r is a quotient of 
n x • • • x r r . Since n ^ 3, Theorem 8.2 implies that it is not cuspidal, which implies that r ^ 2. 
Let k denote the largest integer among the deg(rj)’s and let r,; have degree k with i maximal for 
this property. Then by [14] and Lemma 4.5, one may assume that i = r. Now write r = T r and 
let p be an irreducible subquotient of ri x • • • x r r _i such that 7r is a quotient of p x r. Since it 
is distinguished, sopxr is. Apply Lemma 8.8 to this product. According to Theorem 8.2, we 
obtain that k must be ^ 2. Moreover, if k = 2, then p = 1 n _ 2 . □ 

8.5. Distinction of the twists of A n and II n 

We first determine which twists of A n are distinguished. 

Lemma 8.11. — Let n ^ 2 and x e Gi. Then A n ■ x is distinguished if and only if x = 1. 

Proof. — If / divides n, then A n is the trivial character and the result is immediate. If / does 
not divide n, then we have the exact sequence: 

o -► Vn ■ x -► V ra • X = (j'n-l • x) X U (n+1) / 2 X An ■ X — 0 

By Lemma 8.8 with k = n — 1, the representation V n • x, and hence A n ■ x, is non-distinguished 
for x f {1, }. If x = u” 1 is non-trivial (which forces e > 1), then Lemma 8.7 together with 
Corollary 6.2 imply that A n ■ u -1 is not distinguished. Now assume that x = 1- 

If e > 1, the contragredient V* is distinguished by Lemma 8.8 (A) but v~ l is not. Thus V* 
carries a nonzero invariant linear form vanishing on 1 . It thus gives a nonzero invariant linear 
form on the subrepresentation A*. By Proposition 8.3, the representation A n is distinguished. 

If e = 1, then V n = l n ® A n by Proposition 5.2. By Lemma 8.8, we have d(V n ) ^ 2 since 
Conditions (A) and (B) are fulfilled. Thus A n is distinguished with d(A n ) = d(V n ) — 1. □ 
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Corollary 8.12. - Assume that e > 1. All the irreducible representations of G n , n ^ 3 in the 

list given by Theorem 1.1 are distinguished. 

Proof. — When applied with k = n — 1 and k = n — 2 respectively, Lemma 8.8 gives the result 
for v n \ x y and l n _2 x r. By passing to the contragredient (Proposition 8.3), we get the result 
for the representation r' T ' _ 1 x x when e > 1. 

By Lemma 8.11, A n is distinguished. By passing to the contragredient, we get the result for 
A* when e > 1. (Note that A n is selfdual when e = 1.) This finishes the proof. □ 

We now determine which twists of II n are distinguished. This is done in Lemma 8.11 when / 
does not divide n. We now treat the case where / divides n. 

Lemma 8.13. — Assume that e is not 1 and divides n. For x e Gi, the representations H n ■ x 
and II* • x are not distinguished. 

Proof. — By Proposition 8.3, it is enough to prove it for II* • x- By Lemma 8.7, for II* • x to 
be distinguished, it is necessary that at least one of the derivatives (II* • x)^ for i = 1,2 has a 
character as a quotient. We have: 

( n * • X) (1) = A*_1 • x^- 1/2 , (n* • x) (2) = < l -2 • x- 

By Lemma 8.7, we conclude that II* • x is not distinguished when x^. It remains to consider 
n* • iq or rather its contragredient II n ■ v 1 . Its second derivative is i/ /; 2 . By Lemma 8.7, our 
claim follows. □ 

Lemma 8.If.. - Assume that e = 1 and i divides n. For x e Gi, the representations II n • x is 
distinguished if and only if x = 1 ■ 

Proof. — When e = 1, the representation II n is selfdual thus the first part of the proof of Lemma 
8.13 still holds. Thus Ii n ■ x is not distinguished for any x^ 1. However, the second derivative 
of n n is l n _ 2 , thus Lemma 8.7 is not sufficient to determine whether or not n n is distinguished. 

Let H n act on X = P( n _i i i)\G n . There are two closed orbit A and B in X, where A is reduced 
to a point and B is isomorphic to P( n - 2 ,i)\G n -i (see [19, 5]). Since q is congruent to 1 mod the 
characteristic of R, the modulus R-character of P( n _i p is trivial. By [21, Proposition 1.2.8], 
there is a non-zero G n -invariant linear form px on V n . Similarly, there is a non-zero H n -inva- 
riant linear form on C®(B,R). Composition by the restriction from X to B gives us a non-zero 
H,„-invariant linear form pb on V n . Finally, for / e C®(X, R), we write Pa(/) for the value of / 
at A. We thus get three H n -invariant linear forms on V n . 

The form px is actually G n -equivariant; its image is l n , and its kernel W n has length 2, with 
socle l n (the space of R-valued constant functions on X) and irreducible quotient n n . We claim 
that these three linear forms are linearly independent. Granting the claim, there is no nontrivial 
linear combination of pa, Pb that vanishes on W„. Moreover, if /o denotes the constant function 
taking value 1, and if pb is chosen so that Pb(/o) = 1> then: 

(p A - Pb)(/o) = 0. 

Therefore, pa — Pb is a nonzero H,„-invariant linear forms on W n that vanishes on the space 
of constant functions; it thus induces a nonzero H„-invariant linear forms on H„. Finally H„ is 
H,„-distinguished when e = 1 and t divides n. 
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It remains to prove the claim. Let U denote the unique open H n -orbit in X, so that X is the 
disjoint union of A, B and U, and let U be its preimage in G n . Let p be a Haar measure on G n . 
Since G n is locally pro-p, there is a compact open subset c U with nonzero measure. Write cj) 
for the characteristic function of the image of II in X. By [21, §2.8], there exists a a £ R x such 
that: 

M-x(^) = ol • p(ln) X 0. 

On the other hand, we have Pa(0) = Pb^) = 0 and hence the linear forms px, Pa and Pb are 
linearly independent. □ 

Remark 8.15. — Suppose e = 1 and t does not divide n. It follows from the proof of Lemma 
8.14 that d(V n ) is at least 3. On the other hand, the conditions of Lemma 8.8 implies that there 
is at most one H,„-invariant linear form upto scalars on each of three orbits A, B and U. Thus, 
d(V n ) = 3. Since V n = l n ® II n , it follows that d(II n ) = 2. 

8.6. First reduction of the problem 

Thanks to Corollary 8.10, we are already reduced to studying those H„-distinguished irredu¬ 
cible representations of G n , with n ^ 3, whose cuspidal support is made of characters. 

Lemma 8.16. Let p e be such that pW ■ has a trivial quotient. Then p is 

one of the following representations: 

(1) x P with pe Gi - {i/('»- 2fc - 1 )/ 2 ,i/(»»-i)/2}; 

( 2 ) vt m ; 

(3) A£ • zz( n_fc )/ 2 . 

Proof. — We follow the proof given in the complex case in [19, Lemma 6.2]. The condition on 
p is equivalent to saying that p embeds into a representation of the form: 

V(p) = v { k-i~ k)/2 x p, peGi. 

If p f {i/( n_2fc_1 )/ 2 ; j/( n ~ 1 )/ 2 } ; this representation is irreducible (see Proposition 4.1) thus p is as 
in Case 1. Assume that e > 1. Thanks to Lemma 4.2, Proposition 4.10 and (5.2), we have: 

(1) V(u( n ~ 1 ) //2 ) has a unique irreducible subrepresentation, which is uj" fc ^ 2 . Thus p is as in 
Case 2. 

(2) V(zy( n ^ 2fc ^ 1 )/ 2 ) has a unique irreducible subrepresentation, which is A? • z/ n-fe )/ 2 . Thus p 
is as in Cases 2 or 3. 

Assume now that e = 1. Then, by Proposition 5.2, any subrepresentation p of V(zz( n ~W 2 ) is as 
in Case 2 or 3. Note that, in the case where / divides k, the representation V& is indecomposable, 
thus p must be the character Vjf ' k ^ 2 . This finishes the proof of Lemma 8.16. □ 

In conclusion, we have the following result. 

Proposition 8.17. Assume n ^ 3. Let ir e G n be Jl n -distinguished. Then there are p £ G n _i 
and x £ Gi such that it is an irreducible quotient of p x x and at least one of the following 
conditions holds: 

(1) One has p = u“^{ 2 or p = v][\. 

(2) One has p = A*_ x • u 1 / 2 . 

(3) One has p = l n _2 x p for some p e Gi — {zz _ ( n_1 )/ 2 , i/( n_1 )/ 2 }. 
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(4) The representation p-u 1 / 2 is H n _i- distinguished and x = v 3 A 2 . 

Moreover, if e > 1, then n is the unique irreducible quotient of p x x- 

In order to prove our main theorem 1.1, our strategy is to study, by induction on n 5= 2, the 
irreducible quotients of p x x in all these cases when e > 1, and to prove that they are either in 
the list of Theorem 1.1 or non-distinguished. 

Assuming that Theorem 1.1 holds for G n _i with n ^ 3, we thus have to study the distinction 
of the following representations: 

Case 1: the irreducible quotients of v n ^!{ x x and u n-i x X f° r X 6 Gi; 

Case 2: the irreducible quotients of l n _2 x p x x for p £ Gi — z/O 1-1 )/ 2 }, X e Gi! 

Case 3: the irreducible quotients of A*_ x • v 1//2 x x f° r X e Gi! 

Case 4: the irreducible quotients of p x z/ - ( n-3 )/ 2 where p is: 

(4.a) the character (included in Case 1 above); 

(4.b) a representation l n _2 x p with pe Gi — zA n_1 )/ 2 } (included in Case 2); 

(4.c) a representation v n -2 x with {i/~( n_ 'h/ 2 , i/( n+1 )/ 2 }; 

(4.d) a representation i/ _ 3 x r with re G 2 infinite-dimensional; 

(4.e) one of the representations A n _i • zA/ 2 or A*_ x • 1 A / 2 (see Case 3 above); 

Cases 1 and 4.c are treated in Section 9 for arbitrary e ^ 1, and Case 4.d is treated in Section 
11 for e > 1. 

We reduce Case 2 to studying Q (p x A n _i • n _1//2 ) for pe Gi — {i/ - ( n-1 )/ 2 , ^h 1 - 1 )/ 2 } i n Section 
10, when e > 1. 

In Section 12, we do the remaining cases when e > 1. 


1/2 ^ 

9. Computing the irreducible quotients of j x x for \ e Gi 

Lemma 9.1. Assume e > 1. Let a,b e Z with a ^ b. For x e Gi, write V(x) = Z([a, b]) x x- 

(1) If X f {A 1-1 , v b+1 }, thenV(x) is irreducible. 

(2) Assume that x = v b+l and e does not divide n. Then \{u b+1 ) has length 2 and we have 
the following exact sequence: 

0 — Z([a, b + 1]) — \{u b+1 ) — Z([a, 6] + [6 + 1]) — 0. 

(3) Assume that x = and e does not divide n. Then V(n a_1 ) has length 2 and we have 
the following exact sequence: 

0 -► Z([a, 6] + [a — 1]) -► V(n a " 1 ) -► Z([o - 1,6]) 0. 

(4) //e divides n, then = zA +1 and V(zA +1 ) has length 3 with: 

S(V(*/ +1 )) = Z([a,6 + 1]), Q(V(n b+1 )) = Z([a — 1, b]). 

Proof. — Case 1 follows from Propositions 4.1 and 4.8. The other cases reduce to Proposition 
4.10 by twisting by the character x -1 > since V(x) ■ n _c = Z([a — c, b — c]) x x v ~ c f° r c e Z. □ 

From Lemma 9.1 we get the following proposition. 
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Proposition 9.2. 


Assume e > 1. For all n ^ 1, we have: 

v]l-\ x X */X i {u _(n_1)/2 , u (n+1)/2 }, 


Q(^-i x X) = 1, 

At 


if x = v 1 ^ //2 , 
if X = v( n+1 ^ 2 . 


Twisting by v 1 , we get the following. 

Proposition 9.3. — Assume e > 1. For all n ^ 1, we have: 

”n-l X X if X i {u~ (n+1)/2 , zA”- 1 )/ 2 }, 


Q(^n-( 2 X X) = 


,-l 


ifx= v (n+1)/2 , 


A n • u _1 if x = zA n ” 1 )/ 2 . 

By duality, we get the following. 

Proposition 9.f. - Assume e > 1. For all n ^ 1, we have: 

( V - X J 2 x x ifx f {i/- (n + 1)/2 ,i/ (n - 1)/2 }, 

S(u n _{ xx)= l n if x = u (n ~ 1)/2 , 

l A* ifx = u' (n+1)/2 . 

Twisting by u, we get the following. 

Proposition 9.5. - Assume e > 1. For all n ^ 1, we have: 

"n-l XX */ X ^ {U^ (n_1)/2 , u( n+1 )/ 2 }, 

X = V {U+1)/2 , 

A*'U ifx = Z/-( n - 1)/2 . 

In the case where e = 1, we summarize below the results obtained in Section 5. 
Proposition 9.6. - Assume e = 1. 

(1) If x A zA n+1 )/ 2 ; then x x is irreducible. 

(2) If 0. does not divide n, the irreducible quotients of rfjl x x i y( n+1 )/ 2 are l n and Tl r 

(3) If i divides n, the irreducible quotient of x zA n+1 )/ 2 is l n . 

Thus we have treated Case 1 of Proposition 8.17. 

Corollary 9.7. Let e > 1 and p e Gi — {u (' n 3 )/ 2 , zy( n+1 )/ 2 |_ Then: 

-(n-3)/2x _ f P x v]!\ if n A u" (n_1)/2 , 


S( v n -1 X x) = 


QK-2 X p, X 1/ (n 3)/2 ) = 


A*-u if p = 


Proof. - By assumption on p, the representation v n -2 x p is irreducible. It is thus isomorphic 
to p x u n _ 2 . It thus suffices to consider the representation n(p) = Q (p x u n _2 x zy~( n_3 )/ 2 ). By 
Proposition 9.2, we have: 

QK-2 X U-(- 3 )/ 2 ) = ^ 

1/2 1/2 

thus 7r(/i) is equal to Q(/r x z' n _ ] ). By assumption on p. the representation p x u n _ 1 is reducible 

if and only if p = ^ _ ( n_1 )/ 2 . Finally, the representation: 

7r (l/-(»-l)/2) = Q(^-("- 1 )/ 2 X 1/^j) = S(^ / _ 2 1 X jy-^- 1 )/ 2 ) 
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is equal to A* • n by Proposition 9.3. By Lemma 8.11, it is not distinguished. 
Thus we have treated Case 4.c of Proposition 8.17. 


□ 


10. Computing Q(l n _2 x p x y) for p e G\ — {u ( n P/ 2 , n( n P/ 2 } and y e Gi 

In this section, we fix a character p e Gi different from ^-(^-P/ 2 and zA n- P/ 2 , and we assume 
that e > 1. Note that this implies that l n _2 x p = p x l n _2 is irreducible. For ye Gi, write: 

W(x) = In—2 X p X x- 

We record below two facts in the form of the following lemma which will be used repeatedly in 
what follows. 


Lemma 10.1. — W(y) has a unique irreducible subrepresentation and a unique irreducible 
quotient. Moreover, one has: 


Q(x x X") 


St 2 -x^ 1/2 */e> 2, 

1 2 • A n 1 2 if e = 2. 


Proof. The first statement follows from Lemma 4.2 and the second one from Lemma 9.1. □ 

Lemma 10.2. — For any xf {/in, /in -1 , z/ - ( n- P/ 2 , iA 71-1 )/ 2 }, the representation W(y) is irre¬ 
ducible and distinguished. 


Proof. By Proposition 4.8, W(y) is irreducible. It satifies Condition (A) of Lemma 8.8 with 
k = n — 2, thus it is distinguished. □ 


Lemma 10.3. — One has: 

Q(w(H) 


Q(l„—2 X St 2 • pLV 1 / 2 ) 

Q(ln-2 X 1 2 • /iU _1//2 ) 


and Q(W(/in x )) = Q(l n _2 x 1 2 • /in 1 / 2 ). 


if e > 2, 
if e = 2, 


Proof. — First observe that, by Lemma 10.1, W(/zn) has l n _2 x St 2 - /in 1 / 2 as a quotient if e > 2 
and W(/in -1 ) has l n _2 x I 2 • /in -1 / 2 as a quotient if e ^ 2. Once again, applying Lemma 10.1 
the statement is proved. □ 


Proposition 10.4■ Write Y(/i) = Q(l n _2 x St 2 • /in 1 / 2 ). Then: 


Y(/i) 


l n _2 x St 2 • /in 1 / 2 if /i ^ n ( n+1 )/ 2 or e = 2, 

A* if /i = zy-O+P / 2 and e does not divide n and e > 2 . 


Proof. — The statement follows from Proposition 4.8 if e = 2, and it follows from Proposition 
7.4 if /i 7^ z/ - ( n+1 )/ 2 . Assume that p = z/ - ( n+1 V 2 and e does not divide n and e > 2. We have: 

Y(n -(n+1)/2 ) = Q(W(n -(n-1 Z/ 2 )) 

= Q(n - ( n+1 Z/ 2 x l n _ 2 x n - (- -1 V 2 ) 

- Q(i' _( ” +1)/2 x vTh 

which is equal to A* by applying respectively Lemma 10.3, Lemma 4.5 (since e does not divide n, 
the representation l n _2 x zy~( n+1 )/ 2 is irreducible by Proposition 4.8), Lemma 9.1 and (5.2). □ 
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Proposition 10.5. — Write P(p) = Q(l n _2 
tion P(p) is not distinguished, and we have: 

A 1 / 2 X zy - ( n-3 )/ 2 


P(W (n ~' S)/2 ) = { 

v . n 


x I 2 ■ pu 1//2 ). For p ^ v 3 )/ 2 , the representa- 

if e does not divide n — 2 and e > 2, 
if e = 2 and n is odd. 


Proof. - The first assertion follows form Lemma 8.8. Assume now that p = z/~( n ~ 3 )/ 2 . If e > 2 
does not divide n — 2, then P(i/ - ( n-3 )/ 2 ) = Q(W(u _ ( n_1 ^ 2 ) by Lemma 10.3. By Lemma 9.1 and 
Proposition 4.8, we have: 

Q(W(u- (n " 1)/2 ) = v~ X Jl x i/-( n - 3 )/ 2 . 


Assume now that e = 2 and n is odd. By a similar argument as above, we deduce that: 

P(u — ( n— 3)/ 2 ) = Q(^-(- 3 )/ 2 X V~T{). 

By Proposition 9.4 and the observation following Lemma 4.2, we get P(V - ( n ~ 3 )/ 2 ) = A*. 


□ 


Note that l n _2 x/ix^ = /ix ln -2 x X- Thus: 


Q(W(z>" 1)/2 )) 

Q(W(u“( n - 1 )/ 2 )) 


| Q(p x A n _i • v 1/2 ) 

| Q(M X ^n-f) 

Q(m x ^-l 2 )- 


We have the following proposition. 


if e does not divide n — 1, 
if e divides n — 1, 


Proposition 10.6. - One has: 


Q(W{r-( n ~ 1)/2 )) 


u”Ti 2 x/i ifp^u-^! 2 , 

A * if p = z /-( n+1 )/ 2 and e does not divide n. 


Proof. - This follows from Propositions 9.4 and 9.5. 

It remains to study: 

Q(W(z>" 1)/2 )) = Q (/1 x A n _! • W 1 ' 2 ) 
when e does not divide n — 1. This will be done in Section 12. 


□ 


11. Computing Q(n^ 3 x r x n 3 ^/ 2 ) for re G 2 infinite dimensional 

In this section, we assume that e > 1. We consider all those infinite dimensional r £ G 2 such 
1/2 

that u n _ 3 x r is irreducible, that is: 

(1) r is cuspidal; 

(2) t is a Steinberg representation St 2 • pr 1 ^ 2 with p f {z y~( n-1 V 2 , ^h 1 ^ 1 )/ 2 } an d e > 2; 

(3) t is a principal series A x p with A p~ l f {v~ l ,v} and A ,pf {u"^” 3 )/ 2 , iA™ -1 )/ 2 }. 

In all these cases, we study the unique irreducible quotient: 

(11.1) U(r)=Q(^ s <r»-M' 5 ). 

We first have the following results. 

Lemma 11.1. - For all these r as above, we have U(r) = Q(r x l n _ 2 ). 


32 


V. SECHERRE & C. G. VENKETASUBRAMANIAN 


Proof. — It follows from the fact that i /^ 3 x r = r x z//^ and Q(i/^( 2 3 x u~^ n ~ 3 ^ 2 ) = l n _ 2 - □ 

Proposition 11.2. — Assume that r is cuspidal. Then U(r) = r x l n _ 2 . 

Proof. — This follows from the fact that r x l n _2 is irreducible when r is cuspidal. □ 

We now treat the cases where r is not cuspidal. 

Proposition 11.3. — Assume r = A x p with Xp~ l f {u~ l ,r} and X,pf {z/"( n ~ 3 )/ 2 , z/( n-1 )/ 2 }. 
Then we have: 

U(r) = A x p x l n _2 for all X, p A v~^ n ~ 1 ^ 2 
and, if p = i / _ ( n_1 )/ 2 and e does not divide n — 1, then U(r) is not distinguished. 

Proof. — The first assertion follows from Proposition 4.8. Assume now that p = and 

e does not divide n — 1. It follows from Proposition 9.4 that: 

U(r) = Q(A x A*_! • n 1 / 2 ), 

which is not distinguished by Lemma 8.8 with k = 1. □ 

Proposition 11.4- - Assume e > 2 and r = St 2 • pv 1 / 2 with pf{v ( n 1 )/ 2 ) j/( n 1 )/ 2 }. Then: 

U(r) = r x l n _ 2 for all p A y~^ n+1 ^ 2 
and U(r) is not distinguished for p = z^( n+1 )/ 2 . 

Remark 11.5. If we assume that e = 2 in Lemma 11.4, then r is cuspidal and this case has 
already been done in Lemma 11.2. 

Proof. — Write r = L([0,1]) • p. By Proposition 7.4, the representation r x l n _2 is irreducible 
unless p = v k with k a half-integer and the segments [—(n — 3)/2, (n — 3)/2] and [k, k + 1] are 
juxtaposed, that is p = zA n-1 )/ 2 (which is not allowed) or p = z, - ( n+1 )/ 2 . 

Assume p = zy ~( n+1 )/ 2 and e does not divide n (thus p A u^ n ^ 1 ^ 2 ). Let L be the irreducible 
quotient of St 2 • pv 1 ! 2 x . If e > 3, note that St 3 • v 1 is the unique irreducible quotient 

of St 2 • z / -3 / 2 x 1 (see p. 168 of [15] and the exact sequence (3.5) in [19]). Twisting by i/ - ( n ~ 3 )/ 2 , 
we see that L = St 3 • i/ - ( n-1 )/ 2 . Moreover, by [15, Theorem 2] or [19, Remark 6.7], no twist of 
L is distinguished. If e = 3, L is equal to a twist of II 3 , which is not distinguished by Lemma 
8.13. Hence, no twist of L is distinguished. Applying Lemma 8.8 with k = n — 3 to r ' n / _ 3 x L 
yields that it is not distinguished, and so U(r) is not distinguished. □ 

12. The remaining cases 

In this section, we assume that e > 1 as in Sections 10 and 11. It remains for us to study the 
distinction of the following representations: 

(1) the irreducible quotients of p x A n _i • v~ x ^ 2 for p e Gi — iA”-" 1 )/ 2 }; 

( 2 ) the irreducible quotients of A*_ x • zA / 2 x y for y e Gi; 

(3) the irreducible quotient of A n _i • z / 1//2 x zy _ ( n ~ 3 )/ 2 . 

Note that we may assume e does not divide n — 1 (or else A n _i would be the trivial character). 

The first case is the one that remains from Section 10, the second one corresponds to Case 3 
of Paragraph 8.6 and the third one corresponds to the part of Case 4.e of Paragraph 8.6 which 
does not belong to Case 3. 
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12.1. Distinction of /j x A n _i • v x ! 2 and A*_ x • u 1 / 2 x x 

In this paragraph, we show that, if A*_ x • u 1 / 2 x x is distinguished, then x must be equal to 
3 )/ 2 _ Qj ven this, it will follow by Proposition 8.5 that /i x A n _i • u -1 / 2 is distinguished if 
and only if p = z /( n_3 )/ 2 . 

Lemma 12.1. — Let x e Gi and e > 1. T/ien f/ie representation Sto • u^ 1 / 2 x x is distinguished 
if and only if x = 1- 

Proof. — Write B(x) = St 2 • u ~ 1//2 x x- If X = 1) then B(l) is distinguished as it satisfies (B) of 
Lemma 8.8 for k = 2. 

Assume x f {1, zx, 2 }. Since x is nontrivial, Lemma 8.8 implies that B(x)* is not distingui¬ 
shed. Since x f {u, u~ 2 }, Lemma 7.3 shows that B(x) is irreducible. Thus, by Lemma 8.3, B(x) 
is not distinguished. It remains to consider the case when x e {u, u~ 2 }. 

If e > 3, then we remind that [St 2 • u 3 / 2 x 1 ] = St 3 • v + A 3 as in the complex case (see p. 168 of 
[15] or (3.5) in [19]). First we twist St 2 • u 3 ' 2 x 1 by u~ 2 . Secondly, we take the contragredient 
St 2 • x 1 and twist by v. These yield: 

[B(W 2 )] = A 3 • W 2 + St 3 ■ I/" 1 , [B(u)] = A| • u + St 3 

respectively. None of these subquotients are distinguished. 

If e = 2, then St 2 is cuspidal, thus B(u) is irreducible and the result follows from Lemma 8 . 8 . 
We finally assume that e = 3. We first claim the principal series £ = x 1 x v has length 
7, with subquotients: 

1 3 , u 3 , u^ 1 , n 3 , n 3 • u, II 3 • z / _1 and the cuspidal representation St 3 . 

Indeed, £ contains 1 3 and II 3 as well as their twists by v and u 2 , and it also contains the cuspidal 
(thus nondegenerate) representation St 3 with multiplicity 1. The Jacquet module r( 1;11 )(^) has 
length 6 , thus our claim follows. Now we have: 

[£] » fc 1,! »]+[B(„)] 

- (i 3 +1/3 1 + n 3 ) + [B(0] 

by Proposition 4.10. It follows that: 

[B(u)] = u 3 + II 3 + n 3 • v + St 3 

in the Grothendieck group of finite length representations of G 3 . 

By Lemma 8.13 and Theorem 8.2, none of these subquotients are distinguished. Since B(z/~ 2 ) 
is equal to B(u), our lemma is proved. □ 

Given x e Gi, we now write: 

A(x) = K-i • ^ V2 x X- 

We study the distinction of A(x) in the following lemma. 

Lemma 12.2. — Assume that e does not divide n— 1, and let x e Gi. Then A(x) is distingui¬ 
shed if and only if x = i/ _ ( n ~ 3 )/ 2 . 
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Proof. — First, Lemma 8.8 with k = n — 1 shows that A(u~^ n ~ 3 ^ 2 ) is distinguished. 

For the converse, we may assume that n ^ 4 since we have treated the case when n = 3 in 
Lemma 12.1. Assume first that e > 2. By Proposition 10.4, A(x) is a quotient of: 

dl 2 3 » St, • X X , 

which is distinguished by Remark 8.9 if and only if Condition (A) of Lemma 8.8 is satisfied with 
k = n — 3. This is the case if and only if St 2 • z/ -1 / 2 x ^z/( n ~ 3 )/ 2 j s distinguished. By Lemma 
12.1, this happens if and only if x = zz~( n ~ 3 )/ 2 . 

Assume now that e = 2. Note that the characters zA^ 1 )/ 2 and z/( n+1 )/ 2 are the only ones that 
are obtained from j,^( n ~ 3 )/ 2 up to a translation of an integer power of u. 

Assume first that x f {xx C 77 - -3 )/ 2 , zA n-1 )/ 2 }. Then A(x) is irreducible by Proposition 4.1, and 
Lemma 8.8 implies that A(y)* is not distinguished. By Proposition 8.3, A(x) is not distinguished 
either. 

It remains to consider the case where x = = z/( n+1 )/ 2 . We write A = A(v( n+1 ^ 2 '>). 

By definition, A*_ 1 -v 1 / 2 is the unique irreducible quotient of z/( n+1 )/ 2 x l n _ 2 - The representation 
A is thus a quotient of V = z /( n+1 )/ 2 x l n _2 x z/( n+1 )/ 2 . Now write the two exact sequences: 


0 


-V 2 _ „(n + l)/2 x 


'n -1 


n —2 


A* 

A n -1 


M 2 


0 


(n+l)/2 


y 1/2 

'n—1 


0 . 


( 12 . 1 ) 

and: 

(12.2) 0 - A*_ x • V 2 ' 2 - In —2 X 1 / 

Computing (12.1) x z/( n+1 )/ 2 , we get: 

0 —► W —► V A 0 

where W is the representation x zy( n+1 )/ 2 ; which is irreducible since zv ( ri+1 )/ 2 zy ( n_1 )/ 2 and 

zx'( ra+ i )/ 2 / l /-( n + 1 )/ 2 . Thus W is isomorphic to z /( n+1 )/ 2 x ■ Computing z /( n+1 )/ 2 x (12.2) we 
get: 

,1/2 . w /T 


0 


Z>+l)/2 X A*_! 


V 


W^O. 


Observe that W is distinguished by Lemma 8 . 8 , thus V is also distinguished. Lemma 8.8 (applied 
with k = n — 1) also shows that the space of H n -invariant forms on V is one-dimensional. 

Now we claim A is not distinguished. Assume A is distinguished, and let T denote a nonzero 
invariant linear form on V which is trivial on Ki = Ker(cc). Since V has a one-dimensional space 
of invariant forms, T is proportional to any nonzero invariant linear form on V which is trivial 
on K 2 = Ker(/?). Thus, T is zero on Ki + K 2 . Since T is nonzero, Ki + K 2 is different from the 
whole space V. Since Ki is irreducible and isomorphic to W, we get that Ki + K 2 = K 2 , thus: 


Ki c K 2 ^ z> + 1 V 2 x A*_ x • 


d /2 


It follows that: 


W = S(z > +1)/2 x A*_! • z/ 1 / 2 ) = Q(A). 

Thus W • v is the unique irreducible quotient of A • v. Observe that W • v = x z / n+3 )/ 2 is 
isomorphic to W* and hence is distinguished by Proposition 8.3. However, the representation 
A ■ v = A *_ 1 • i / -1 / 2 x z /( n_1 )/ 2 is not distinguished by Lemma 8 . 8 , a contradiction. □ 
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12.2. Distinction of Q(A*_ X • u 1 / 2 x u ( n 3 )/ 2 ) and Q (z/ n 3 )/ 2 x A n _i • v 1 / 2 ) 

By Lemma 12.2, in order to finish Cases 1 and 2 of Section 12 for e > 1, it remains to discuss 
the distinction of the irreducible quotients: 

Q(A*_ X • u 1 / 2 x j,-^- 3 )/ 2 ) and Q(*>- 3 )/ 2 x A n _i • u~ 1 / 2 ). 

Note that the latter is the contragredient of the former, thus it is enough to study the distinction 
of the first one. Moreover, if n = 3, then: 

Q(A£ • v 1 / 2 x 1) = St 2 • u _1/2 x 1 

is distinguished by Lemmas 7.3 and 12 . 1 . So we will assume that n ^ 4 in the remainder of this 
Section. In what follows, the computation of distinguished quotients will fall into three cases: 

( 1 ) e > 2 and e does not divide n — 2 ; 

( 2 ) e > 2 and e divides n — 2 (this implies that e does not divide n); 

(3) e = 2 (this implies that e divides n — 2 since e does not divide n — 1). 

We start with the following lemma, which follows from Lemma 9.1. 


Lemma 12.3. — Assume e > 1. We have: 


[v~-? x u" (n_3)/2 ] = < 


l/ n ^ 2 X V ( n 3 )/ 2 

if e> 2 

and 

1 n A II n • V 

if e > 2 

and 

v n X + 

if e = 2 

and 

. in + 1 + 

if e = 2 

and 


We now define two irreducible representations of G n . 
Definition 12.f. Assume e > 1 and n ^ 4. Define: 


$n = Z 


= Z 


( 

n — 

3 

n — 

3 

+ 

n — 1 

n — 

3 


2 


2 


2 ’ 

2 


( 

n — 

3 

n — 

3 

+ 

n + 1 

n — 

r 

\ 

2 


2 


2 ’ 

2 



Observe that <3? n is selfdual if e divides n — 2 and 'L ri is selfdual if e divides n. We also recall 
that II* = n n • z / _1 if e divides n. 


Lemma 12.5. Assume e > 1 and n 5= 4, and suppose e does not divide n— 1. The irreducible 
subquotients of: 

(12.3) 1 „_ 2 X 1 2 • 2)/2 = z ^ 

are: 

( 1 ) the representations x i y-O -3 )/ 2 and <h n if e does not divide n — 2, 

(2) the representations l n , II* • v, II n • and & n if e divides n~ 2. 

Moreover, all subquotients appear with multiplicity 1 if e > 2. If e = 2, only l n may appear with 
multiplicity more than 1. 


n — 3 n — 3 


x Z 


n —- 1 n — 3 
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Proof. — We apply Proposition 4.13. The irreducible subquotients 4> n and: 


n — 1 n — 3 


+ 

CO 

1 

g 

1_ 

^ 1 

2 

i 1 



[ v n }/\ xi/ (" 3 1/ 2 if e does not divide n — 2 , 


n n • ^ 1 


if e divides n — 2 , 


always occur in (12.3). The irreducible subquotients Z([—(n — 3)/2, (n —1)/2] + [—(n —1)/2]) and 
Z([—(n — l)/2, (n — l)/2]) = l n occur if and only if e divides n — 2. The irreducible subquotients 
Z([—(n — 3)/2, (n + l)/2]) = u n and Z([—(n + l)/2, (n — 3)/2]) = uif 1 do not occur, since e does 
not divide n — 1 and e > 1 . □ 

Similarly, by applying Proposition 4.13, we have the following. 

Lemma 12.6. - Assume e > 1 and n 5= 4, and suppose e does not divide n — 1. T/ie irreducible 

subquotients of l n -2 x I 2 • zx -n / 2 are: 

( 1 ) f/ie representations v n and i/e does not divide n, 

( 2 ) f/ie representations v n , and if e divides n. 

Moreover, all subquotients appear with multiplicity 1 if e > 2. If e = 2, only v n may appear with 
multiplicity more than 1. 


Lemma 12.7. 


Assume e does not divide n — 2 nor n — 1, thus e > 2. Then: 
Q(A*_ X • v 1 / 2 x r,-(™- 3 )/ 2 ) = l n _2 x St 2 • i/-^" 2 )/ 2 . 


Proof. — Since e does not divide n — 2 , the product l n -2 x ^ - ( n-3 )/ 2 is irreducible, thus it is 
isomorphic to i /~( n_3 )/ 2 x l n - 2 - Moreover, the representation ^ _ ( n_1 )/ 2 x j ,-!™- 3 )/ 2 x \ n _ 2 has a 
unique irreducible quotient by Proposition 4.2. It follows that this unique irreducible quotient 
is St 2 • ^ _ ( n_2 )/ 2 x l n _ 2 , which is irreducible by Proposition 7.4. □ 

Lemma 12.8. - Assume e > 1 and n ^ 4, and suppose e does not divide n — 1. If the repre¬ 
sentation Q(A*_ X • v 1 ! 2 x i/M n ~ 3 )/ 2 ) distinguished, then it is either l n or l n -2 x St 2 • u~^ n ~ 2 ^ 2 . 

Proof. — If e > 2 and does not divide n — 2 we reduce to the case of Lemma 12.7. Therefore, we 
need only consider either e = 2 or e divides n — 2. The representation B = A *_ 1 • v 1 / 2 x i/~( n ~ 3 )/ 2 
is a quotient of U = i /~( n-1 )/ 2 x l n _ 2 x i/~( n ~ 3 )/ 2 . Observe that we have [U] = [P] + [B] where 
P = v n ^{ x z,-( n - 3 )/ 2 . Now U has the same semisimplification as l n -2 x z,~( n_ 1 V 2 x ^-(n- 3 )/ 2 ^ 
thus we have: 

In —2 x St 2 • z /-( n “ 2 )/ 2 + [ln _2 x 1 2 • ^/M™- 2 )/ 2 ] if e > 2, 

ln -2 X St 2 • 2)/2 + [l n _2 X 1 2 • 2 )/2] + [l n _ 2 X 1 2 ■ V~ n t 2 \ if e = 2, 


[U] = 


since l n -2 x St 2 • v ( n 2 ^ 2 is irreducible by Proposition 7.5. Since e does not divide n — 1, the 
irreducible subquotients occurring in l n -2 x 1 2 • j / _ ( n ~ 2 )/ 2 by Lemma 12.5 are: 

^ v ■ On ^n ’ ^ i ^n ’ ^5 

Moreover, all of them occur with multiplicity 1 except l n , which may appear with larger multi¬ 
plicity if e = 2. Also, By Lemma 12.6, since e does not divide n — 1, the irreducible subquotients 
occurring in l n -2 x 1 2 ■ y~ n ' 2 are v n , and ’L n . Here 4/ n always occurs with multiplicity one 
and if e = 2 the other factors might appear with larger multiplicity. We will now obtain [B] by 
comparing [U] obtained from the two different expressions for [U] above. 
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Assume that e > 2 and e divides n — 2. Then by Lemma 12.3 we have [P] = l n + II n ■ v 1 . 
Hence we have: 

[B] = + n* + l n _ 2 x St 2 • j,-^- 2 )/ 2 . 

Next assume that e = 2. Then e necessarily divides n — 2 (since e does not divide n — 1). By 
Lemma 12.3, we have [P] = l n + u n + n*. Recall that if e divides n then n* = n n ■ u _1 . Hence 
the only possible irreducible subquotients of B are: 

In, »n, In— 2 X St 2 • V ~^~ 2 ')l 2 , , n* • V, <F n , tf n . 

The proof of Lemma 12.8 will be complete if we prove the following lemma. 

Lemma 12.9. Assume e > 1 and n ^ 4, and suppose that e does not divide n — 1. For any 
character y e Gi, f/ie twists • y and 41 n • y ore not distinguished. 

Proof. — Observe that <h n -y and T n -y have only first and second derivatives which are nonzero. 
Thus we will use Lemma 8.7. 

Assume the first derivative of <h n • x has a quotient isomorphic to zz ^ . By Lemma 8.16, this 
would imply that <h n • y is a character, or that the multisegment that corresponds to it is made 
of one segment of length n — 1 and one of length 1, which is not the case. The same argument 
holds for • y. 

From Lemma 12.6, we see that the second derivative of 'J/ n • x is {y n -{ x v~^ n+1 ^ 2 ) ■ y, and 
since e does not divide n — 1 it is irreducible for all y £ Gi by Lemma 9.1. Thus it does not 
have any character as a quotient. Now we have: 

[In—2 X 1 2 ■ , 2 -<- 2 >/ 2 ]P> = [„„V 2 X , 2 -<'->/ 2 ] 

( + 1„_2 + n 5-2 if e divides n - 2 . 

\ zz“l 2 + n *_ 2 if e does not divide n — 2 . 

By Lemma 6.1 and Corollary 6.3, we have (R n • zz -1 / 2 ) = u n -2 and (n*-iz/ 2) = l n _ 2 . Therefore, 
we conclude using Lemma 12.5 that the second derivative of <h n is n*_ 2 . By Lemma 8.7, <L n • y 
and • y are not distinguished. □ 

This ends the proof of Lemma 12.8. □ 

12.3. Distinction of Q(A n _i • zz 1//2 x i/ - ( n-3 )/ 2 ) 

We begin this paragraph with a simple lemma which we will need in the sequel. We remind 
that n ^ 4 and e does not divide n — 1. 

Lemma 12.10. — Let n ^ 4. Assume that e > 1 and /ef A,/r £ Gi — {zz _ ( n “ 3 )/ 2 }. T/ien fde 
induced representation l n _ 2 x A x /r das a unique irreducible quotient. 

Proof. — If A = //, the result follows from [12, Lemma 6.1]. We thus assume that A//i. By 
the geometric lemma, the semi-simplification of the Jacquet module T*( n _ 2 , 1 , 1 )(ln _2 x A x p) is 
the sum of the following representations: 

( 1 ) l n _ 2 (g) A (x)/q 

( 2 ) 1 „_ 2 ® q (x) A, 

(3) [zT/ 3 2 x A] ® zz ( n - 3 )/ 2 ® q, 

(4) K // 2 x q] ® u (- 3 )/ 2 <g> A, 
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( 5 ) 3)/2 , 

(6) \^n-3 x h] <8> A (g) z/( n - 3 )/ 2 , 

(7) X A X /i] ® i/( n ~ 5 )/ 2 0 j/^- 3 )/ 2 ^ 

in the Grothendieck group of finite length representations of the Levi subgroup G ra _2 xGjxGi. 
If A, fi A i/( n ~ 3 )/ 2 then by [ 13 , Lemme 2.4] the representation l n _2 xAx/i has a unique irreducible 
subrepresentation. The result follows by taking contragredients. □ 

Lemma 12.11. — Assume that e > 2 and e does not divide n — 2. Then: 

Q(A n _! • X !/-("-= 3 >/ 2 ) = A n . 

Proof. — The representation C = A ra _i • z^ 1 / 2 x ^-(”-3)/2 j g a q UO tient of: 

W = V n —2 x z> +1)/2 x z.-( n - 3 )/ 2 . 

If we apply Lemma 12.10 with A = z^ 71-1 )/ 2 and g, = j/ - ( n_1 )/ 2 . which is possible since e > 2 and 
e does not divide n — 2, we deduce that W-z/ _1 (thus W) has a unique irreducible quotient. Since 
z/(” +1 )/ 2 x i/~( n_3 )/ 2 is irreducible, it is isomorphic to z, - ( n-3 )/ 2 x zA n+1 )/ 2 . Thus x z/ n+1 )/ 2 
is a quotient of W, and it has the unique irreducible quotient A n . □ 

Lemma 12.12. — Assume that e > 1 and n ^ 4. // Q(A n _i • z/ 1 / 2 x j,-("- 3 )/ 2 ) distinguished, 
then it is A n . 


Proof. — If e > 2 and does not divide n — 2 we reduce to the case of Lemma 12.11. We may 
assume that e = 2 or e divides n — 2. In this proof, W, C are as in Lemma 12.11 and U, P are 
as in Lemma 12.8. Assume that e divides n — 2. Then z /( n_1 )/ 2 = z / - ( n-3 )/ 2 and therefore 

W • z ,- 1 = l n _2 x x z/ -(n-l)/ 2 _ 


Therefore, we have 

[W] = [U • i/] and [W] = [P* • v\ + [C] 
where P * ■ v = v^-i x z/ n-1 )/ 2 . 

If e > 2 and e divides n — 2, then we twist the subquotients of U in the proof of Lemma 12.8 
by v to get: 

[w] = v n ~2 x st 2 • i /- (n " 4)/2 + • z/ + n n + n n • z , 2 + Vn 

and [P* • v] = v n + U n • v 2 . It follows that: 


[C] = v n -2 x st 2 • z ,-(- 4 )/ 2 + • Z/ + n n . 


Hence the only distinguished subquotient is Yl n which is the definition of A n when e does not 
divide n. 

Now e = 2, which necessarily divides n — 2. Then P* • v is isomorphic to P. We twist the 
subquotients of U in the proof of Lemma 12.8 by v to conclude that the only possible irreducible 
subquotients of W are: 

iraj ■ n n , n n , • z/, Tjj ■ z/ 

with all representations except possibly l n and v n appearing with multiplicity greater than 1 . 
Since [P] = l n + u n + n* it follows that the only possible irreducible subquotients of C are: 


1 m ,J n ■ n n , • V, • V. 


Hence the only distinguished subquotient is l n which is the definition of A n when e divides n. 
This completes the proof of the Lemma. □ 
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Remark 12.13. — Given n e G n , we write d(ir) for the dimension of HoniH ri ('/r, R). We notice 
that, unlike in the complex case [1], when R is of positive characteristic, it is not true in general 
that d(ir) < 1 for all n e G n . For example, when n = 2 and e = 1, we have constructed n e G n 
with d(j r) = 2 (see Theorem 3.5). More generally, when e = 1 and l divides n, we have shown 
(see Remark 8.15) that d(n n ) = 2. However, when e > 1 we expect that d(n) ^ 1 for all ir e G n , 
as in the case n = 2. It is interesting to note the analogy of the situation in the case of e = 1 of 
Theorem 3.5 with [ 17 , Corollary 3.3], where the author shows that d(n) ^ 2 where G = GL ra (F ? ) 
and R is an algebraically closed field of characteristic prime to 2 and q. 
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